-----============= acceptance-small: conf-sanity ============----- Fri Dec 20 01:14:16 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg444-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg444-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg444-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg444-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg444-server oleg444-server: oleg444-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg444-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server == conf-sanity test 45: long unlink handling in ptlrpcd == 01:15:30 (1734675330) start mds service on oleg444-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg444-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 PASS 45 (146s) == conf-sanity test 69: replace an OST with the same index ========================================================== 01:17:56 (1734675476) start mds service on oleg444-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1497 (time 1734675523.96 total 10.00 last 149.69) - open/close 2879 (time 1734675533.96 total 20.01 last 138.13) - open/close 4295 (time 1734675543.97 total 30.01 last 141.51) - open/close 5772 (time 1734675553.98 total 40.02 last 147.59) - open/close 7283 (time 1734675563.98 total 50.02 last 151.09) - open/close 8646 (time 1734675573.98 total 60.02 last 136.27) total: 10000 open/close in 69.31 seconds: 144.27 ops/second - unlinked 0 (time 1734675584 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1167 (time 1734675636.17 total 10.00 last 116.66) - open/close 2407 (time 1734675646.18 total 20.01 last 123.92) - open/close 3712 (time 1734675656.18 total 30.01 last 130.44) - open/close 5141 (time 1734675666.19 total 40.02 last 142.82) - open/close 6514 (time 1734675676.19 total 50.02 last 137.27) - open/close 7840 (time 1734675686.19 total 60.02 last 132.59) - open/close 9330 (time 1734675696.20 total 70.03 last 148.91) total: 10000 open/close in 75.27 seconds: 132.86 ops/second - unlinked 0 (time 1734675703 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1501 (time 1734675758.18 total 10.00 last 150.06) - open/close 2813 (time 1734675768.19 total 20.01 last 131.08) - open/close 4143 (time 1734675778.19 total 30.01 last 132.98) - open/close 5467 (time 1734675788.19 total 40.01 last 132.39) - open/close 6672 (time 1734675798.20 total 50.02 last 120.43) - open/close 7958 (time 1734675808.20 total 60.02 last 128.58) - open/close 9260 (time 1734675818.23 total 70.05 last 129.80) total: 10000 open/close in 76.49 seconds: 130.73 ops/second - unlinked 0 (time 1734675826 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1249 (time 1734675881.05 total 10.01 last 124.80) - open/close 2605 (time 1734675891.05 total 20.01 last 135.55) - open/close 3963 (time 1734675901.06 total 30.02 last 135.69) - open/close 5383 (time 1734675911.06 total 40.02 last 141.99) - open/close 6742 (time 1734675921.07 total 50.03 last 135.76) - open/close 7949 (time 1734675931.08 total 60.04 last 120.64) - open/close 9188 (time 1734675941.08 total 70.04 last 123.87) total: 10000 open/close in 75.95 seconds: 131.66 ops/second - unlinked 0 (time 1734675948 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1776 (time 1734675995.85 total 10.01 last 177.44) - open/close 3444 (time 1734676005.86 total 20.01 last 166.74) - open/close 4928 (time 1734676015.86 total 30.01 last 148.37) - open/close 6424 (time 1734676025.86 total 40.02 last 149.53) - open/close 7971 (time 1734676035.87 total 50.02 last 154.67) - open/close 9688 (time 1734676045.87 total 60.02 last 171.68) total: 10000 open/close in 61.91 seconds: 161.52 ops/second - unlinked 0 (time 1734676048 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1648 (time 1734676088.89 total 10.00 last 164.77) - open/close 3013 (time 1734676098.90 total 20.01 last 136.42) - open/close 4389 (time 1734676108.90 total 30.01 last 137.55) - open/close 5646 (time 1734676118.90 total 40.01 last 125.69) - open/close 6868 (time 1734676128.91 total 50.02 last 122.15) - open/close 8291 (time 1734676138.91 total 60.02 last 142.27) - open/close 9620 (time 1734676148.91 total 70.02 last 132.83) total: 10000 open/close in 73.28 seconds: 136.46 ops/second - unlinked 0 (time 1734676154 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1537 (time 1734676204.30 total 10.00 last 153.67) - open/close 3899 (time 1734676214.31 total 20.01 last 236.09) - open/close 5949 (time 1734676224.31 total 30.01 last 204.98) - open/close 8055 (time 1734676234.31 total 40.01 last 210.57) total: 10000 open/close in 47.36 seconds: 211.15 ops/second - unlinked 0 (time 1734676242 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1736 (time 1734676280.54 total 10.00 last 173.55) - open/close 3671 (time 1734676290.54 total 20.00 last 193.46) - open/close 5866 (time 1734676300.54 total 30.01 last 219.43) - open/close 7489 (time 1734676310.54 total 40.01 last 162.28) - open/close 9438 (time 1734676320.54 total 50.01 last 194.86) total: 10000 open/close in 52.84 seconds: 189.26 ops/second - unlinked 0 (time 1734676324 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1937 (time 1734676362.96 total 10.00 last 193.65) - open/close 3770 (time 1734676372.96 total 20.00 last 183.29) - open/close 5400 (time 1734676382.96 total 30.01 last 162.93) - open/close 6981 (time 1734676392.96 total 40.01 last 158.09) - open/close 9194 (time 1734676402.97 total 50.01 last 221.25) total: 10000 open/close in 54.73 seconds: 182.70 ops/second - unlinked 0 (time 1734676409 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1680 (time 1734676450.02 total 10.00 last 167.96) - open/close 3592 (time 1734676460.02 total 20.01 last 191.13) - open/close 5847 (time 1734676470.02 total 30.01 last 225.48) - open/close 7604 (time 1734676480.02 total 40.01 last 175.60) - open/close 9141 (time 1734676490.03 total 50.01 last 153.69) total: 10000 open/close in 55.46 seconds: 180.30 ops/second - unlinked 0 (time 1734676496 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg444-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 PASS 69 (1097s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 01:36:13 (1734676573) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg444-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg444-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 oleg444-server: oleg444-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1342 sec ========== 01:36:39 (1734676599) === conf-sanity: start cleanup 01:36:39 (1734676599) === === conf-sanity: finish cleanup 01:36:41 (1734676601) ===