-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 22:10:34 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg135-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg135-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server oleg135-server: oleg135-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server == conf-sanity test 45: long unlink handling in ptlrpcd == 22:12:38 (1746065558) start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg135-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 PASS 45 (187s) == conf-sanity test 69: replace an OST with the same index ========================================================== 22:15:46 (1746065746) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 882 (time 1746065815.21 total 10.01 last 88.14) - open/close 1730 (time 1746065825.23 total 20.02 last 84.68) - open/close 2559 (time 1746065835.23 total 30.02 last 82.89) - open/close 3407 (time 1746065845.24 total 40.03 last 84.72) - open/close 4454 (time 1746065855.24 total 50.03 last 104.66) - open/close 5436 (time 1746065865.24 total 60.04 last 98.15) - open/close 6529 (time 1746065875.25 total 70.04 last 109.26) - open/close 7415 (time 1746065885.25 total 80.04 last 88.59) - open/close 8273 (time 1746065895.26 total 90.05 last 85.75) - open/close 9092 (time 1746065905.26 total 100.05 last 81.86) - open/close 9918 (time 1746065915.27 total 110.06 last 82.52) total: 10000 open/close in 110.85 seconds: 90.21 ops/second - unlinked 0 (time 1746065918 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1017 (time 1746065995.18 total 10.00 last 101.66) - open/close 1942 (time 1746066005.19 total 20.01 last 92.42) - open/close 2840 (time 1746066015.19 total 30.02 last 89.75) - open/close 3758 (time 1746066025.20 total 40.02 last 91.77) - open/close 4508 (time 1746066035.20 total 50.02 last 74.99) - open/close 5242 (time 1746066045.20 total 60.03 last 73.36) - open/close 5913 (time 1746066055.21 total 70.03 last 67.06) - open/close 6647 (time 1746066065.21 total 80.04 last 73.36) - open/close 7309 (time 1746066075.22 total 90.04 last 66.19) - open/close 8063 (time 1746066085.22 total 100.04 last 75.37) - open/close 8828 (time 1746066095.23 total 110.05 last 76.45) - open/close 9557 (time 1746066105.24 total 120.06 last 72.81) total: 10000 open/close in 126.11 seconds: 79.29 ops/second - unlinked 0 (time 1746066114 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 651 (time 1746066207.23 total 10.01 last 65.03) - open/close 1338 (time 1746066217.25 total 20.03 last 68.57) - open/close 2088 (time 1746066227.27 total 30.06 last 74.79) - open/close 2826 (time 1746066237.28 total 40.06 last 73.73) - open/close 3533 (time 1746066247.29 total 50.07 last 70.65) - open/close 4307 (time 1746066257.30 total 60.09 last 77.30) - open/close 5248 (time 1746066267.32 total 70.10 last 93.98) - open/close 6147 (time 1746066277.32 total 80.10 last 89.85) - open/close 6931 (time 1746066287.32 total 90.11 last 78.38) - open/close 7654 (time 1746066297.34 total 100.12 last 72.19) - open/close 8427 (time 1746066307.36 total 110.14 last 77.16) - open/close 9154 (time 1746066317.37 total 120.15 last 72.59) - open/close 9954 (time 1746066327.39 total 130.17 last 79.90) total: 10000 open/close in 130.71 seconds: 76.50 ops/second - unlinked 0 (time 1746066330 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 815 (time 1746066418.12 total 10.01 last 81.43) - open/close 1610 (time 1746066428.12 total 20.01 last 79.48) - open/close 2434 (time 1746066438.12 total 30.01 last 82.39) - open/close 3239 (time 1746066448.12 total 40.02 last 80.48) - open/close 4083 (time 1746066458.13 total 50.02 last 84.37) - open/close 4995 (time 1746066468.14 total 60.03 last 91.10) - open/close 5730 (time 1746066478.16 total 70.05 last 73.38) - open/close 6583 (time 1746066488.16 total 80.05 last 85.25) - open/close 7387 (time 1746066498.16 total 90.06 last 80.37) - open/close 8135 (time 1746066508.17 total 100.06 last 74.76) - open/close 9005 (time 1746066518.19 total 110.08 last 86.87) - open/close 9775 (time 1746066528.19 total 120.08 last 76.95) total: 10000 open/close in 122.60 seconds: 81.56 ops/second - unlinked 0 (time 1746066533 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 937 (time 1746066621.92 total 10.01 last 93.57) - open/close 1809 (time 1746066631.94 total 20.03 last 87.05) - open/close 2626 (time 1746066641.95 total 30.04 last 81.65) - open/close 3498 (time 1746066651.95 total 40.04 last 87.18) - open/close 4429 (time 1746066661.95 total 50.04 last 93.08) - open/close 5300 (time 1746066671.96 total 60.05 last 87.05) - open/close 6096 (time 1746066681.96 total 70.05 last 79.56) - open/close 6983 (time 1746066691.96 total 80.05 last 88.70) - open/close 7762 (time 1746066701.96 total 90.05 last 77.87) - open/close 8501 (time 1746066711.98 total 100.07 last 73.78) - open/close 9263 (time 1746066721.98 total 110.07 last 76.19) - open/close 9968 (time 1746066732.01 total 120.10 last 70.31) total: 10000 open/close in 120.47 seconds: 83.01 ops/second - unlinked 0 (time 1746066735 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 893 (time 1746066822.97 total 10.00 last 89.26) - open/close 1712 (time 1746066832.98 total 20.01 last 81.82) - open/close 2704 (time 1746066842.99 total 30.03 last 99.07) - open/close 3577 (time 1746066852.99 total 40.03 last 87.29) - open/close 4531 (time 1746066863.00 total 50.03 last 95.39) - open/close 5441 (time 1746066873.01 total 60.04 last 90.88) - open/close 6313 (time 1746066883.01 total 70.05 last 87.16) - open/close 7352 (time 1746066893.02 total 80.05 last 103.84) - open/close 8395 (time 1746066903.02 total 90.06 last 104.27) - open/close 9275 (time 1746066913.03 total 100.06 last 87.94) total: 10000 open/close in 108.61 seconds: 92.07 ops/second - unlinked 0 (time 1746066924 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 771 (time 1746067013.61 total 10.00 last 77.07) - open/close 1504 (time 1746067023.62 total 20.02 last 73.22) - open/close 2308 (time 1746067033.63 total 30.03 last 80.29) - open/close 3151 (time 1746067043.64 total 40.03 last 84.25) - open/close 3917 (time 1746067053.64 total 50.03 last 76.60) - open/close 4678 (time 1746067063.67 total 60.07 last 75.84) - open/close 5393 (time 1746067073.68 total 70.08 last 71.45) - open/close 6227 (time 1746067083.68 total 80.08 last 83.37) - open/close 7144 (time 1746067093.69 total 90.09 last 91.61) - open/close 7979 (time 1746067103.69 total 100.09 last 83.49) - open/close 8720 (time 1746067113.70 total 110.10 last 74.04) - open/close 9427 (time 1746067123.71 total 120.11 last 70.64) total: 10000 open/close in 127.22 seconds: 78.61 ops/second - unlinked 0 (time 1746067133 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 1048 (time 1746067215.41 total 10.01 last 104.74) - open/close 2108 (time 1746067225.42 total 20.01 last 105.92) - open/close 3152 (time 1746067235.42 total 30.02 last 104.38) - open/close 4079 (time 1746067245.43 total 40.02 last 92.64) - open/close 4973 (time 1746067255.43 total 50.03 last 89.34) - open/close 5863 (time 1746067265.44 total 60.03 last 88.97) - open/close 6742 (time 1746067275.44 total 70.03 last 87.89) - open/close 7619 (time 1746067285.44 total 80.03 last 87.70) - open/close 8574 (time 1746067295.44 total 90.04 last 95.45) - open/close 9386 (time 1746067305.45 total 100.04 last 81.17) total: 10000 open/close in 106.62 seconds: 93.79 ops/second - unlinked 0 (time 1746067314 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 879 (time 1746067389.96 total 10.00 last 87.87) - open/close 1731 (time 1746067399.96 total 20.01 last 85.16) - open/close 2633 (time 1746067409.96 total 30.01 last 90.20) - open/close 3590 (time 1746067419.97 total 40.01 last 95.65) - open/close 4489 (time 1746067429.98 total 50.02 last 89.81) - open/close 5457 (time 1746067439.98 total 60.02 last 96.79) - open/close 6462 (time 1746067449.98 total 70.03 last 100.45) - open/close 7369 (time 1746067459.98 total 80.03 last 90.70) - open/close 8394 (time 1746067469.99 total 90.04 last 102.42) - open/close 9432 (time 1746067479.99 total 100.04 last 103.78) total: 10000 open/close in 105.52 seconds: 94.77 ops/second - unlinked 0 (time 1746067487 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1205 (time 1746067552.34 total 10.00 last 120.44) - open/close 2322 (time 1746067562.34 total 20.01 last 111.63) - open/close 3166 (time 1746067572.34 total 30.01 last 84.39) - open/close 4058 (time 1746067582.35 total 40.02 last 89.15) - open/close 4959 (time 1746067592.37 total 50.04 last 89.93) - open/close 5881 (time 1746067602.37 total 60.04 last 92.18) - open/close 6828 (time 1746067612.38 total 70.05 last 94.57) - open/close 7688 (time 1746067622.38 total 80.05 last 85.99) - open/close 8544 (time 1746067632.41 total 90.08 last 85.38) - open/close 9385 (time 1746067642.41 total 100.08 last 84.09) total: 10000 open/close in 108.19 seconds: 92.43 ops/second - unlinked 0 (time 1746067653 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg135-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre On OST0, 10709 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 PASS 69 (2015s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 22:49:21 (1746067761) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 oleg135-server: oleg135-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2364 sec ========== 22:49:59 (1746067799) === conf-sanity: start cleanup 22:50:00 (1746067800) === === conf-sanity: finish cleanup 22:50:01 (1746067801) ===