-----============= acceptance-small: conf-sanity ============----- Thu May 1 19:16:53 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg424-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg424-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg424-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server == conf-sanity test 45: long unlink handling in ptlrpcd == 19:18:08 (1746141488) start mds service on oleg424-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg424-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 PASS 45 (140s) == conf-sanity test 69: replace an OST with the same index ========================================================== 19:20:28 (1746141628) start mds service on oleg424-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1122 (time 1746141671.73 total 10.01 last 112.14) - open/close 2984 (time 1746141681.73 total 20.01 last 186.15) - open/close 4266 (time 1746141691.73 total 30.01 last 128.19) - open/close 5435 (time 1746141701.74 total 40.01 last 116.84) - open/close 6678 (time 1746141711.74 total 50.02 last 124.25) - open/close 7862 (time 1746141721.75 total 60.02 last 118.34) - open/close 8944 (time 1746141731.75 total 70.02 last 108.18) total: 10000 open/close in 78.90 seconds: 126.75 ops/second - unlinked 0 (time 1746141741 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1449 (time 1746141806.23 total 10.00 last 144.85) - open/close 2791 (time 1746141816.23 total 20.01 last 134.18) - open/close 3964 (time 1746141826.24 total 30.01 last 117.25) - open/close 5228 (time 1746141836.24 total 40.01 last 126.38) - open/close 6473 (time 1746141846.24 total 50.02 last 124.41) - open/close 7540 (time 1746141856.26 total 60.03 last 106.57) - open/close 8588 (time 1746141866.27 total 70.04 last 104.70) - open/close 9677 (time 1746141876.27 total 80.04 last 108.87) total: 10000 open/close in 83.18 seconds: 120.22 ops/second - unlinked 0 (time 1746141881 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1002 (time 1746141947.79 total 10.00 last 100.20) - open/close 2035 (time 1746141957.80 total 20.01 last 103.23) - open/close 3174 (time 1746141967.81 total 30.01 last 113.82) - open/close 4265 (time 1746141977.81 total 40.01 last 109.09) - open/close 5476 (time 1746141987.81 total 50.02 last 121.03) - open/close 6881 (time 1746141997.83 total 60.03 last 140.32) - open/close 8139 (time 1746142007.83 total 70.04 last 125.73) - open/close 9479 (time 1746142017.84 total 80.04 last 133.94) total: 10000 open/close in 83.91 seconds: 119.17 ops/second - unlinked 0 (time 1746142023 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1328 (time 1746142080.73 total 10.00 last 132.78) - open/close 2549 (time 1746142090.73 total 20.00 last 122.07) - open/close 3702 (time 1746142100.73 total 30.01 last 115.29) - open/close 4812 (time 1746142110.73 total 40.01 last 111.00) - open/close 6192 (time 1746142120.75 total 50.02 last 137.84) - open/close 7502 (time 1746142130.75 total 60.02 last 130.97) - open/close 8790 (time 1746142140.75 total 70.02 last 128.79) - open/close 9981 (time 1746142150.76 total 80.03 last 119.03) total: 10000 open/close in 80.13 seconds: 124.80 ops/second - unlinked 0 (time 1746142152 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1143 (time 1746142213.79 total 10.01 last 114.19) - open/close 3121 (time 1746142223.79 total 20.01 last 197.75) - open/close 4694 (time 1746142233.80 total 30.02 last 157.17) - open/close 5868 (time 1746142243.80 total 40.02 last 117.38) - open/close 7025 (time 1746142253.81 total 50.03 last 115.65) - open/close 8245 (time 1746142263.81 total 60.03 last 121.99) - open/close 9439 (time 1746142273.81 total 70.03 last 119.38) total: 10000 open/close in 74.54 seconds: 134.15 ops/second - unlinked 0 (time 1746142280 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1261 (time 1746142329.60 total 10.01 last 125.98) - open/close 2493 (time 1746142339.61 total 20.01 last 123.14) - open/close 4031 (time 1746142349.61 total 30.02 last 153.71) - open/close 5385 (time 1746142359.62 total 40.02 last 135.33) - open/close 6691 (time 1746142369.63 total 50.03 last 130.51) - open/close 8033 (time 1746142379.63 total 60.03 last 134.17) - open/close 9239 (time 1746142389.64 total 70.05 last 120.47) total: 10000 open/close in 77.32 seconds: 129.33 ops/second - unlinked 0 (time 1746142398 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1296 (time 1746142456.98 total 10.00 last 129.60) - open/close 2940 (time 1746142466.99 total 20.00 last 164.38) - open/close 4737 (time 1746142476.99 total 30.00 last 179.69) - open/close 6109 (time 1746142486.99 total 40.01 last 137.12) - open/close 7538 (time 1746142496.99 total 50.01 last 142.86) - open/close 8845 (time 1746142507.00 total 60.01 last 130.68) total: 10000 open/close in 69.50 seconds: 143.89 ops/second - unlinked 0 (time 1746142518 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 991 (time 1746142581.19 total 10.00 last 99.05) - open/close 2203 (time 1746142591.19 total 20.01 last 121.14) - open/close 3386 (time 1746142601.20 total 30.02 last 118.16) - open/close 4475 (time 1746142611.20 total 40.02 last 108.88) - open/close 5629 (time 1746142621.21 total 50.03 last 115.37) - open/close 6840 (time 1746142631.21 total 60.03 last 121.01) - open/close 8210 (time 1746142641.22 total 70.03 last 136.98) - open/close 9482 (time 1746142651.22 total 80.04 last 127.09) total: 10000 open/close in 83.42 seconds: 119.87 ops/second - unlinked 0 (time 1746142656 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1248 (time 1746142703.67 total 10.01 last 124.73) - open/close 2536 (time 1746142713.69 total 20.02 last 128.60) - open/close 3793 (time 1746142723.69 total 30.02 last 125.66) - open/close 5013 (time 1746142733.70 total 40.03 last 121.93) - open/close 6003 (time 1746142743.70 total 50.03 last 98.98) - open/close 7011 (time 1746142753.70 total 60.04 last 100.76) - open/close 8112 (time 1746142763.71 total 70.04 last 110.05) - open/close 9362 (time 1746142773.72 total 80.05 last 124.89) total: 10000 open/close in 85.52 seconds: 116.93 ops/second - unlinked 0 (time 1746142781 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1114 (time 1746142854.44 total 10.00 last 111.38) - open/close 2263 (time 1746142864.45 total 20.01 last 114.84) - open/close 3391 (time 1746142874.45 total 30.01 last 112.79) - open/close 4548 (time 1746142884.45 total 40.01 last 115.69) - open/close 5959 (time 1746142894.45 total 50.01 last 141.06) - open/close 7184 (time 1746142904.46 total 60.02 last 122.43) - open/close 8404 (time 1746142914.46 total 70.02 last 121.94) - open/close 9329 (time 1746142924.47 total 80.03 last 92.46) total: 10000 open/close in 85.77 seconds: 116.59 ops/second - unlinked 0 (time 1746142932 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg424-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 PASS 69 (1394s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:43:43 (1746143023) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1641 sec ========== 19:44:15 (1746143055) === conf-sanity: start cleanup 19:44:16 (1746143056) === === conf-sanity: finish cleanup 19:44:17 (1746143057) ===