-----============= acceptance-small: conf-sanity ============----- Thu May 1 20:19:32 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg408-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server == conf-sanity test 45: long unlink handling in ptlrpcd == 20:20:54 (1746145254) start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 45 (132s) == conf-sanity test 69: replace an OST with the same index ========================================================== 20:23:06 (1746145386) start mds service on oleg408-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2345 (time 1746145425.20 total 10.00 last 234.45) - open/close 4527 (time 1746145435.20 total 20.01 last 218.13) - open/close 6585 (time 1746145445.21 total 30.01 last 205.67) - open/close 8160 (time 1746145455.21 total 40.01 last 157.47) - open/close 9775 (time 1746145465.21 total 50.01 last 161.48) total: 10000 open/close in 51.20 seconds: 195.32 ops/second - unlinked 0 (time 1746145467 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1253 (time 1746145503.93 total 10.00 last 125.28) - open/close 3219 (time 1746145513.93 total 20.00 last 196.60) - open/close 4815 (time 1746145523.93 total 30.00 last 159.60) - open/close 6653 (time 1746145533.94 total 40.01 last 183.69) - open/close 8245 (time 1746145543.94 total 50.01 last 159.15) - open/close 9641 (time 1746145553.95 total 60.02 last 139.46) total: 10000 open/close in 63.04 seconds: 158.63 ops/second - unlinked 0 (time 1746145558 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1349 (time 1746145621.33 total 10.00 last 134.85) - open/close 2674 (time 1746145631.34 total 20.01 last 132.44) - open/close 3869 (time 1746145641.34 total 30.01 last 119.48) - open/close 5090 (time 1746145651.34 total 40.01 last 122.08) - open/close 6307 (time 1746145661.34 total 50.02 last 121.64) - open/close 7537 (time 1746145671.35 total 60.03 last 122.89) - open/close 8670 (time 1746145681.36 total 70.03 last 113.28) total: 10000 open/close in 79.97 seconds: 125.04 ops/second - unlinked 0 (time 1746145692 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1329 (time 1746145743.17 total 10.00 last 132.86) - open/close 2889 (time 1746145753.17 total 20.00 last 155.99) - open/close 4352 (time 1746145763.17 total 30.01 last 146.23) - open/close 5989 (time 1746145773.18 total 40.01 last 163.60) - open/close 7201 (time 1746145783.18 total 50.02 last 121.17) - open/close 9163 (time 1746145793.18 total 60.02 last 196.18) total: 10000 open/close in 64.91 seconds: 154.05 ops/second - unlinked 0 (time 1746145799 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1690 (time 1746145846.70 total 10.00 last 168.98) - open/close 3805 (time 1746145856.71 total 20.00 last 211.45) - open/close 5791 (time 1746145866.71 total 30.01 last 198.56) - open/close 7398 (time 1746145876.71 total 40.01 last 160.69) - open/close 8958 (time 1746145886.71 total 50.01 last 155.95) total: 10000 open/close in 56.15 seconds: 178.08 ops/second - unlinked 0 (time 1746145894 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1468 (time 1746145948.83 total 10.00 last 146.76) - open/close 2688 (time 1746145958.84 total 20.01 last 121.85) - open/close 3750 (time 1746145968.85 total 30.02 last 106.15) - open/close 5061 (time 1746145978.86 total 40.03 last 130.97) - open/close 6512 (time 1746145988.86 total 50.03 last 145.04) - open/close 7812 (time 1746145998.87 total 60.04 last 129.90) - open/close 8842 (time 1746146008.87 total 70.04 last 103.00) - open/close 9990 (time 1746146018.87 total 80.05 last 114.74) total: 10000 open/close in 80.13 seconds: 124.80 ops/second - unlinked 0 (time 1746146020 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1265 (time 1746146078.77 total 10.01 last 126.37) - open/close 2657 (time 1746146088.78 total 20.02 last 139.08) - open/close 3876 (time 1746146098.79 total 30.02 last 121.85) - open/close 5336 (time 1746146108.80 total 40.03 last 145.88) - open/close 6676 (time 1746146118.80 total 50.03 last 133.95) - open/close 7863 (time 1746146128.80 total 60.04 last 118.65) - open/close 8859 (time 1746146138.80 total 70.04 last 99.58) - open/close 9916 (time 1746146148.81 total 80.04 last 105.67) total: 10000 open/close in 80.72 seconds: 123.89 ops/second - unlinked 0 (time 1746146151 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1373 (time 1746146202.07 total 10.00 last 137.30) - open/close 2633 (time 1746146212.07 total 20.01 last 125.92) - open/close 4308 (time 1746146222.08 total 30.01 last 167.41) - open/close 5464 (time 1746146232.08 total 40.02 last 115.56) - open/close 6936 (time 1746146242.09 total 50.02 last 147.15) - open/close 8247 (time 1746146252.09 total 60.02 last 131.04) - open/close 9377 (time 1746146262.10 total 70.03 last 112.94) total: 10000 open/close in 75.19 seconds: 133.00 ops/second - unlinked 0 (time 1746146269 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1633 (time 1746146321.46 total 10.00 last 163.24) - open/close 2744 (time 1746146331.46 total 20.01 last 111.07) - open/close 3856 (time 1746146341.47 total 30.01 last 111.17) - open/close 5110 (time 1746146351.47 total 40.01 last 125.37) - open/close 6363 (time 1746146361.47 total 50.01 last 125.28) - open/close 7430 (time 1746146371.47 total 60.02 last 106.67) - open/close 8648 (time 1746146381.48 total 70.02 last 121.73) - open/close 9578 (time 1746146391.49 total 80.03 last 92.94) total: 10000 open/close in 83.60 seconds: 119.62 ops/second - unlinked 0 (time 1746146396 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1531 (time 1746146453.06 total 10.00 last 153.02) - open/close 2904 (time 1746146463.07 total 20.01 last 137.22) - open/close 4223 (time 1746146473.07 total 30.02 last 131.79) - open/close 5536 (time 1746146483.08 total 40.02 last 131.22) - open/close 6649 (time 1746146493.09 total 50.03 last 111.24) - open/close 7923 (time 1746146503.09 total 60.03 last 127.35) - open/close 9174 (time 1746146513.09 total 70.03 last 125.10) total: 10000 open/close in 77.78 seconds: 128.57 ops/second - unlinked 0 (time 1746146522 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.108@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 69 (1228s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:43:34 (1746146614) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1470 sec ========== 20:44:03 (1746146643) === conf-sanity: start cleanup 20:44:04 (1746146644) === === conf-sanity: finish cleanup 20:44:05 (1746146645) ===