-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:00:13 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg208-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg208-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg208-server oleg208-server: oleg208-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:01:31 (1746136891) start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 PASS 45 (128s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:03:39 (1746137019) start mds service on oleg208-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1868 (time 1746137059.23 total 10.00 last 186.78) - open/close 3652 (time 1746137069.23 total 20.00 last 178.37) - open/close 5323 (time 1746137079.24 total 30.01 last 167.05) - open/close 6645 (time 1746137089.24 total 40.01 last 132.14) - open/close 7917 (time 1746137099.24 total 50.01 last 127.18) - open/close 9373 (time 1746137109.25 total 60.02 last 145.46) total: 10000 open/close in 65.08 seconds: 153.66 ops/second - unlinked 0 (time 1746137115 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 2134 (time 1746137164.62 total 10.00 last 213.38) - open/close 4132 (time 1746137174.63 total 20.01 last 199.64) - open/close 6226 (time 1746137184.63 total 30.01 last 209.29) - open/close 8057 (time 1746137194.64 total 40.02 last 183.07) total: 10000 open/close in 49.07 seconds: 203.78 ops/second - unlinked 0 (time 1746137205 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1211 (time 1746137251.00 total 10.00 last 121.09) - open/close 2799 (time 1746137261.01 total 20.01 last 158.61) - open/close 4385 (time 1746137271.01 total 30.02 last 158.49) - open/close 6003 (time 1746137281.02 total 40.02 last 161.78) - open/close 7783 (time 1746137291.02 total 50.02 last 177.94) - open/close 9382 (time 1746137301.02 total 60.03 last 159.87) total: 10000 open/close in 63.53 seconds: 157.42 ops/second - unlinked 0 (time 1746137305 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1652 (time 1746137352.70 total 10.00 last 165.17) - open/close 3217 (time 1746137362.70 total 20.01 last 156.37) - open/close 4729 (time 1746137372.71 total 30.02 last 151.07) - open/close 6308 (time 1746137382.71 total 40.02 last 157.87) - open/close 8292 (time 1746137392.72 total 50.02 last 198.32) total: 10000 open/close in 59.47 seconds: 168.16 ops/second - unlinked 0 (time 1746137403 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1973 (time 1746137448.98 total 10.01 last 197.19) - open/close 3891 (time 1746137458.98 total 20.01 last 191.79) - open/close 5283 (time 1746137468.99 total 30.01 last 139.11) - open/close 6775 (time 1746137479.00 total 40.02 last 149.08) - open/close 8317 (time 1746137489.00 total 50.02 last 154.16) - open/close 9887 (time 1746137499.00 total 60.02 last 157.00) total: 10000 open/close in 60.56 seconds: 165.12 ops/second - unlinked 0 (time 1746137500 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1635 (time 1746137542.61 total 10.01 last 163.41) - open/close 3521 (time 1746137552.62 total 20.01 last 188.51) - open/close 5027 (time 1746137562.62 total 30.02 last 150.52) - open/close 6458 (time 1746137572.63 total 40.02 last 142.99) - open/close 7783 (time 1746137582.63 total 50.03 last 132.47) - open/close 9399 (time 1746137592.63 total 60.03 last 161.60) total: 10000 open/close in 64.49 seconds: 155.06 ops/second - unlinked 0 (time 1746137598 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1592 (time 1746137650.92 total 10.00 last 159.13) - open/close 2882 (time 1746137660.92 total 20.00 last 129.00) - open/close 4630 (time 1746137670.93 total 30.01 last 174.76) - open/close 6617 (time 1746137680.93 total 40.01 last 198.63) - open/close 9039 (time 1746137690.93 total 50.01 last 242.11) total: 10000 open/close in 55.52 seconds: 180.13 ops/second - unlinked 0 (time 1746137697 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1314 (time 1746137748.00 total 10.01 last 131.29) - open/close 2415 (time 1746137758.01 total 20.02 last 109.97) - open/close 3491 (time 1746137768.01 total 30.02 last 107.57) - open/close 4723 (time 1746137778.01 total 40.02 last 123.19) - open/close 5866 (time 1746137788.01 total 50.02 last 114.29) - open/close 6935 (time 1746137798.02 total 60.03 last 106.85) - open/close 8169 (time 1746137808.03 total 70.04 last 123.29) - open/close 9346 (time 1746137818.03 total 80.04 last 117.64) total: 10000 open/close in 85.08 seconds: 117.54 ops/second - unlinked 0 (time 1746137824 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1460 (time 1746137877.21 total 10.00 last 145.98) - open/close 2857 (time 1746137887.22 total 20.01 last 139.64) - open/close 4563 (time 1746137897.22 total 30.01 last 170.50) - open/close 6414 (time 1746137907.23 total 40.02 last 185.03) - open/close 8102 (time 1746137917.23 total 50.02 last 168.68) - open/close 9644 (time 1746137927.24 total 60.03 last 154.14) total: 10000 open/close in 62.59 seconds: 159.76 ops/second - unlinked 0 (time 1746137931 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1828 (time 1746137981.69 total 10.01 last 182.66) - open/close 3644 (time 1746137991.70 total 20.01 last 181.56) - open/close 5682 (time 1746138001.70 total 30.01 last 203.72) - open/close 7060 (time 1746138011.73 total 40.04 last 137.44) - open/close 8744 (time 1746138021.73 total 50.04 last 168.34) total: 10000 open/close in 57.69 seconds: 173.34 ops/second - unlinked 0 (time 1746138030 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.108@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 PASS 69 (1095s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 18:21:54 (1746138114) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 oleg208-server: oleg208-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1332 sec ========== 18:22:26 (1746138146) === conf-sanity: start cleanup 18:22:27 (1746138147) === === conf-sanity: finish cleanup 18:22:28 (1746138148) ===