-----============= acceptance-small: conf-sanity ============----- Tue Oct 1 02:30:21 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg159-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server == conf-sanity test 45: long unlink handling in ptlrpcd == 02:31:31 (1727764291) start mds service on oleg159-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg159-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server unloading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 PASS 45 (143s) == conf-sanity test 69: replace an OST with the same index ========================================================== 02:33:54 (1727764434) start mds service on oleg159-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1161 (time 1727764478.95 total 10.01 last 116.01) - open/close 2409 (time 1727764488.95 total 20.01 last 124.73) - open/close 3695 (time 1727764498.96 total 30.02 last 128.52) - open/close 4824 (time 1727764508.96 total 40.02 last 112.84) - open/close 6114 (time 1727764518.97 total 50.03 last 128.96) - open/close 7411 (time 1727764528.97 total 60.03 last 129.69) - open/close 8496 (time 1727764538.98 total 70.04 last 108.40) - open/close 9699 (time 1727764548.98 total 80.04 last 120.27) total: 10000 open/close in 82.57 seconds: 121.11 ops/second - unlinked 0 (time 1727764553 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1055 (time 1727764618.31 total 10.00 last 105.50) - open/close 2243 (time 1727764628.32 total 20.01 last 118.67) - open/close 3420 (time 1727764638.34 total 30.03 last 117.49) - open/close 4549 (time 1727764648.35 total 40.03 last 112.86) - open/close 5640 (time 1727764658.36 total 50.05 last 108.94) - open/close 6760 (time 1727764668.36 total 60.05 last 111.99) - open/close 7770 (time 1727764678.36 total 70.05 last 101.00) - open/close 8958 (time 1727764688.37 total 80.05 last 118.74) total: 10000 open/close in 89.09 seconds: 112.25 ops/second - unlinked 0 (time 1727764699 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1095 (time 1727764763.32 total 10.00 last 109.50) - open/close 2306 (time 1727764773.33 total 20.01 last 120.99) - open/close 3380 (time 1727764783.35 total 30.03 last 107.14) - open/close 4372 (time 1727764793.37 total 40.05 last 99.06) - open/close 5488 (time 1727764803.38 total 50.06 last 111.49) - open/close 6714 (time 1727764813.38 total 60.06 last 122.55) - open/close 7759 (time 1727764823.39 total 70.07 last 104.45) - open/close 8892 (time 1727764833.39 total 80.07 last 113.28) total: 10000 open/close in 89.35 seconds: 111.92 ops/second - unlinked 0 (time 1727764844 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1120 (time 1727764911.75 total 10.00 last 111.98) - open/close 2093 (time 1727764921.75 total 20.00 last 97.27) - open/close 3116 (time 1727764931.76 total 30.01 last 102.26) - open/close 4149 (time 1727764941.76 total 40.01 last 103.26) - open/close 5270 (time 1727764951.76 total 50.01 last 112.10) - open/close 6303 (time 1727764961.77 total 60.02 last 103.23) - open/close 7398 (time 1727764971.78 total 70.03 last 109.39) - open/close 8584 (time 1727764981.78 total 80.03 last 118.58) - open/close 9710 (time 1727764991.78 total 90.03 last 112.56) total: 10000 open/close in 92.58 seconds: 108.01 ops/second - unlinked 0 (time 1727764996 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1026 (time 1727765058.56 total 10.00 last 102.57) - open/close 2107 (time 1727765068.56 total 20.01 last 108.03) - open/close 3324 (time 1727765078.57 total 30.01 last 121.67) - open/close 4355 (time 1727765088.57 total 40.01 last 103.08) - open/close 5420 (time 1727765098.59 total 50.03 last 106.30) - open/close 6531 (time 1727765108.59 total 60.04 last 111.06) - open/close 7687 (time 1727765118.60 total 70.04 last 115.51) - open/close 8779 (time 1727765128.60 total 80.05 last 109.17) - open/close 9949 (time 1727765138.61 total 90.05 last 116.90) total: 10000 open/close in 90.56 seconds: 110.43 ops/second - unlinked 0 (time 1727765141 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1233 (time 1727765201.73 total 10.01 last 123.22) - open/close 2325 (time 1727765211.74 total 20.01 last 109.15) - open/close 3364 (time 1727765221.74 total 30.02 last 103.82) - open/close 4428 (time 1727765231.76 total 40.03 last 106.28) - open/close 5517 (time 1727765241.76 total 50.03 last 108.86) - open/close 6541 (time 1727765251.76 total 60.04 last 102.37) - open/close 7616 (time 1727765261.77 total 70.04 last 107.42) - open/close 8662 (time 1727765271.77 total 80.05 last 104.58) - open/close 9717 (time 1727765281.78 total 90.06 last 105.39) total: 10000 open/close in 92.84 seconds: 107.71 ops/second - unlinked 0 (time 1727765286 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1216 (time 1727765346.90 total 10.00 last 121.56) - open/close 2339 (time 1727765356.90 total 20.01 last 112.24) - open/close 3511 (time 1727765366.91 total 30.01 last 117.17) - open/close 4820 (time 1727765376.91 total 40.01 last 130.85) - open/close 6133 (time 1727765386.92 total 50.02 last 131.21) - open/close 7591 (time 1727765396.92 total 60.03 last 145.70) - open/close 9395 (time 1727765406.93 total 70.03 last 180.33) total: 10000 open/close in 74.51 seconds: 134.22 ops/second - unlinked 0 (time 1727765413 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1167 (time 1727765467.41 total 10.01 last 116.64) - open/close 2358 (time 1727765477.42 total 20.01 last 119.04) - open/close 3414 (time 1727765487.42 total 30.01 last 105.56) - open/close 4470 (time 1727765497.42 total 40.01 last 105.59) - open/close 5490 (time 1727765507.43 total 50.02 last 101.93) - open/close 6596 (time 1727765517.43 total 60.02 last 110.59) - open/close 7602 (time 1727765527.44 total 70.03 last 100.51) - open/close 8585 (time 1727765537.45 total 80.04 last 98.16) - open/close 9770 (time 1727765547.45 total 90.05 last 118.49) total: 10000 open/close in 91.89 seconds: 108.83 ops/second - unlinked 0 (time 1727765550 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1203 (time 1727765613.21 total 10.00 last 120.26) - open/close 2268 (time 1727765623.24 total 20.03 last 106.22) - open/close 3331 (time 1727765633.24 total 30.03 last 106.29) - open/close 4424 (time 1727765643.24 total 40.04 last 109.25) - open/close 5409 (time 1727765653.24 total 50.04 last 98.49) - open/close 6446 (time 1727765663.25 total 60.04 last 103.64) - open/close 7471 (time 1727765673.25 total 70.04 last 102.48) - open/close 8646 (time 1727765683.25 total 80.05 last 117.46) - open/close 9733 (time 1727765693.26 total 90.05 last 108.65) total: 10000 open/close in 92.73 seconds: 107.84 ops/second - unlinked 0 (time 1727765698 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1020 (time 1727765764.39 total 10.00 last 101.96) - open/close 2044 (time 1727765774.41 total 20.02 last 102.21) - open/close 3040 (time 1727765784.41 total 30.02 last 99.58) - open/close 4047 (time 1727765794.42 total 40.03 last 100.64) - open/close 5084 (time 1727765804.43 total 50.04 last 103.64) - open/close 6106 (time 1727765814.43 total 60.04 last 102.16) - open/close 7182 (time 1727765824.44 total 70.06 last 107.44) - open/close 8081 (time 1727765834.45 total 80.06 last 89.82) - open/close 9043 (time 1727765844.46 total 90.07 last 96.12) total: 10000 open/close in 99.02 seconds: 100.98 ops/second - unlinked 0 (time 1727765855 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server unloading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 PASS 69 (1527s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 02:59:22 (1727765962) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1778 sec ========== 02:59:59 (1727765999) === conf-sanity: start cleanup 03:00:00 (1727766000) === === conf-sanity: finish cleanup 03:00:00 (1727766000) ===