-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 21:54:42 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg209-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg209-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg209-server oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server == conf-sanity test 45: long unlink handling in ptlrpcd == 21:56:21 (1746064581) start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg209-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: rmmod: ERROR: Module mdd is in use pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 PASS 45 (137s) == conf-sanity test 69: replace an OST with the same index ========================================================== 21:58:38 (1746064718) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg209-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1065 (time 1746064762.77 total 10.00 last 106.50) - open/close 2116 (time 1746064772.77 total 20.00 last 105.07) - open/close 3201 (time 1746064782.77 total 30.01 last 108.47) - open/close 4366 (time 1746064792.78 total 40.01 last 116.43) - open/close 5513 (time 1746064802.78 total 50.01 last 114.69) - open/close 6851 (time 1746064812.79 total 60.02 last 133.70) - open/close 8290 (time 1746064822.79 total 70.02 last 143.84) - open/close 9457 (time 1746064832.80 total 80.03 last 116.59) total: 10000 open/close in 84.47 seconds: 118.38 ops/second - unlinked 0 (time 1746064839 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1387 (time 1746064901.73 total 10.01 last 138.56) - open/close 2572 (time 1746064911.73 total 20.01 last 118.49) - open/close 3736 (time 1746064921.73 total 30.01 last 116.37) - open/close 5034 (time 1746064931.74 total 40.02 last 129.70) - open/close 6191 (time 1746064941.74 total 50.03 last 115.64) - open/close 7447 (time 1746064951.75 total 60.03 last 125.59) - open/close 8732 (time 1746064961.75 total 70.04 last 128.38) total: 10000 open/close in 79.17 seconds: 126.31 ops/second - unlinked 0 (time 1746064972 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1238 (time 1746065031.28 total 10.00 last 123.76) - open/close 2306 (time 1746065041.29 total 20.01 last 106.74) - open/close 3362 (time 1746065051.29 total 30.02 last 105.52) - open/close 4456 (time 1746065061.30 total 40.02 last 109.32) - open/close 5629 (time 1746065071.30 total 50.03 last 117.28) - open/close 6746 (time 1746065081.30 total 60.03 last 111.70) - open/close 7987 (time 1746065091.31 total 70.03 last 124.04) - open/close 9343 (time 1746065101.31 total 80.04 last 135.52) total: 10000 open/close in 85.14 seconds: 117.45 ops/second - unlinked 0 (time 1746065107 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1161 (time 1746065167.83 total 10.00 last 116.08) - open/close 2219 (time 1746065177.83 total 20.00 last 105.80) - open/close 3434 (time 1746065187.84 total 30.01 last 121.45) - open/close 4670 (time 1746065197.85 total 40.02 last 123.42) - open/close 5753 (time 1746065207.86 total 50.03 last 108.24) - open/close 6944 (time 1746065217.86 total 60.03 last 119.05) - open/close 8099 (time 1746065227.86 total 70.03 last 115.48) - open/close 9428 (time 1746065237.86 total 80.03 last 132.88) total: 10000 open/close in 83.79 seconds: 119.35 ops/second - unlinked 0 (time 1746065243 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1432 (time 1746065293.83 total 10.00 last 143.17) - open/close 2926 (time 1746065303.83 total 20.00 last 149.36) - open/close 4078 (time 1746065313.84 total 30.01 last 115.11) - open/close 5430 (time 1746065323.84 total 40.01 last 135.19) - open/close 6621 (time 1746065333.84 total 50.01 last 119.09) - open/close 7837 (time 1746065343.86 total 60.03 last 121.38) - open/close 9030 (time 1746065353.87 total 70.04 last 119.25) total: 10000 open/close in 78.89 seconds: 126.77 ops/second - unlinked 0 (time 1746065364 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1171 (time 1746065419.54 total 10.01 last 117.01) - open/close 2506 (time 1746065429.54 total 20.01 last 133.49) - open/close 3861 (time 1746065439.55 total 30.02 last 135.39) - open/close 5032 (time 1746065449.56 total 40.02 last 117.02) - open/close 6120 (time 1746065459.56 total 50.02 last 108.79) - open/close 7389 (time 1746065469.56 total 60.03 last 126.88) - open/close 8641 (time 1746065479.57 total 70.04 last 125.06) - open/close 9812 (time 1746065489.57 total 80.04 last 117.07) total: 10000 open/close in 81.78 seconds: 122.28 ops/second - unlinked 0 (time 1746065493 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1450 (time 1746065554.09 total 10.00 last 144.95) - open/close 2798 (time 1746065564.10 total 20.01 last 134.72) - open/close 4102 (time 1746065574.11 total 30.02 last 130.30) - open/close 5380 (time 1746065584.11 total 40.02 last 127.75) - open/close 6665 (time 1746065594.12 total 50.03 last 128.43) - open/close 8202 (time 1746065604.12 total 60.03 last 153.69) - open/close 9642 (time 1746065614.12 total 70.03 last 143.96) total: 10000 open/close in 72.46 seconds: 138.00 ops/second - unlinked 0 (time 1746065618 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1261 (time 1746065680.19 total 10.00 last 126.09) - open/close 2584 (time 1746065690.19 total 20.00 last 132.27) - open/close 3924 (time 1746065700.19 total 30.01 last 133.97) - open/close 5147 (time 1746065710.19 total 40.01 last 122.28) - open/close 6381 (time 1746065720.20 total 50.01 last 123.37) - open/close 7600 (time 1746065730.20 total 60.01 last 121.87) - open/close 8768 (time 1746065740.22 total 70.03 last 116.61) total: 10000 open/close in 78.89 seconds: 126.76 ops/second - unlinked 0 (time 1746065750 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1100 (time 1746065806.01 total 10.01 last 109.93) - open/close 2293 (time 1746065816.03 total 20.02 last 119.15) - open/close 3305 (time 1746065826.04 total 30.03 last 101.07) - open/close 4391 (time 1746065836.04 total 40.03 last 108.57) - open/close 5352 (time 1746065846.06 total 50.05 last 95.93) - open/close 6393 (time 1746065856.07 total 60.06 last 104.04) - open/close 7413 (time 1746065866.07 total 70.06 last 101.97) - open/close 8374 (time 1746065876.08 total 80.07 last 96.02) - open/close 9422 (time 1746065886.09 total 90.08 last 104.63) total: 10000 open/close in 95.99 seconds: 104.18 ops/second - unlinked 0 (time 1746065894 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1511 (time 1746065958.08 total 10.01 last 151.01) - open/close 2888 (time 1746065968.08 total 20.01 last 137.65) - open/close 4107 (time 1746065978.08 total 30.01 last 121.89) - open/close 5442 (time 1746065988.09 total 40.02 last 133.41) - open/close 6782 (time 1746065998.09 total 50.02 last 133.96) - open/close 7987 (time 1746066008.09 total 60.02 last 120.50) - open/close 9467 (time 1746066018.10 total 70.03 last 147.92) total: 10000 open/close in 73.84 seconds: 135.43 ops/second - unlinked 0 (time 1746066023 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.109@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg209-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre On OST0, 6533 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 PASS 69 (1397s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 22:21:56 (1746066116) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 oleg209-server: oleg209-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1669 sec ========== 22:22:31 (1746066151) === conf-sanity: start cleanup 22:22:32 (1746066152) === === conf-sanity: finish cleanup 22:22:34 (1746066154) ===