-----============= acceptance-small: conf-sanity ============----- Thu May 1 20:09:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg241-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg241-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server == conf-sanity test 45: long unlink handling in ptlrpcd == 20:10:34 (1746144634) start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 PASS 45 (134s) == conf-sanity test 69: replace an OST with the same index ========================================================== 20:12:48 (1746144768) start mds service on oleg241-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1342 (time 1746144808.89 total 10.00 last 134.16) - open/close 2627 (time 1746144818.90 total 20.01 last 128.43) - open/close 4071 (time 1746144828.90 total 30.01 last 144.37) - open/close 5622 (time 1746144838.91 total 40.02 last 154.97) - open/close 7088 (time 1746144848.91 total 50.02 last 146.53) - open/close 8465 (time 1746144858.92 total 60.03 last 137.61) - open/close 9604 (time 1746144868.92 total 70.03 last 113.86) total: 10000 open/close in 73.21 seconds: 136.59 ops/second - unlinked 0 (time 1746144874 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1341 (time 1746144931.74 total 10.00 last 134.09) - open/close 2696 (time 1746144941.75 total 20.01 last 135.43) - open/close 4164 (time 1746144951.75 total 30.01 last 146.80) - open/close 5552 (time 1746144961.75 total 40.01 last 138.79) - open/close 6865 (time 1746144971.75 total 50.01 last 131.28) - open/close 8104 (time 1746144981.75 total 60.01 last 123.86) - open/close 9484 (time 1746144991.76 total 70.01 last 137.96) total: 10000 open/close in 74.17 seconds: 134.83 ops/second - unlinked 0 (time 1746144997 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1270 (time 1746145053.50 total 10.00 last 126.96) - open/close 2654 (time 1746145063.50 total 20.00 last 138.39) - open/close 3912 (time 1746145073.51 total 30.01 last 125.71) - open/close 5366 (time 1746145083.52 total 40.01 last 145.34) - open/close 7102 (time 1746145093.52 total 50.02 last 173.57) - open/close 8609 (time 1746145103.52 total 60.02 last 150.63) - open/close 9995 (time 1746145113.52 total 70.02 last 138.56) total: 10000 open/close in 70.05 seconds: 142.75 ops/second - unlinked 0 (time 1746145115 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1494 (time 1746145162.39 total 10.00 last 149.38) - open/close 2927 (time 1746145172.39 total 20.00 last 143.30) - open/close 4511 (time 1746145182.40 total 30.01 last 158.31) - open/close 6457 (time 1746145192.40 total 40.01 last 194.58) - open/close 8055 (time 1746145202.40 total 50.01 last 159.75) - open/close 9598 (time 1746145212.40 total 60.01 last 154.26) total: 10000 open/close in 63.38 seconds: 157.78 ops/second - unlinked 0 (time 1746145217 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1184 (time 1746145269.76 total 10.00 last 118.40) - open/close 2496 (time 1746145279.77 total 20.02 last 130.97) - open/close 3743 (time 1746145289.78 total 30.02 last 124.64) - open/close 4858 (time 1746145299.79 total 40.03 last 111.43) - open/close 6129 (time 1746145309.79 total 50.04 last 127.01) - open/close 7554 (time 1746145319.80 total 60.04 last 142.44) - open/close 8900 (time 1746145329.80 total 70.04 last 134.55) total: 10000 open/close in 77.30 seconds: 129.36 ops/second - unlinked 0 (time 1746145338 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1358 (time 1746145392.07 total 10.01 last 135.61) - open/close 3059 (time 1746145402.07 total 20.02 last 170.00) - open/close 4343 (time 1746145412.08 total 30.02 last 128.35) - open/close 5482 (time 1746145422.08 total 40.03 last 113.81) - open/close 6677 (time 1746145432.09 total 50.04 last 119.43) - open/close 7894 (time 1746145442.10 total 60.04 last 121.62) - open/close 9244 (time 1746145452.10 total 70.04 last 135.00) total: 10000 open/close in 78.14 seconds: 127.97 ops/second - unlinked 0 (time 1746145462 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 954 (time 1746145522.82 total 10.01 last 95.34) - open/close 1745 (time 1746145532.82 total 20.02 last 79.03) - open/close 2633 (time 1746145542.83 total 30.02 last 88.79) - open/close 3510 (time 1746145552.83 total 40.02 last 87.67) - open/close 4497 (time 1746145562.83 total 50.02 last 98.67) - open/close 5635 (time 1746145572.83 total 60.02 last 113.79) - open/close 6849 (time 1746145582.84 total 70.03 last 121.34) - open/close 8064 (time 1746145592.84 total 80.03 last 121.49) - open/close 8944 (time 1746145602.84 total 90.03 last 87.99) - open/close 9934 (time 1746145612.84 total 100.03 last 98.99) total: 10000 open/close in 100.74 seconds: 99.26 ops/second - unlinked 0 (time 1746145615 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1106 (time 1746145677.05 total 10.00 last 110.59) - open/close 2247 (time 1746145687.06 total 20.00 last 114.09) - open/close 3312 (time 1746145697.06 total 30.00 last 106.49) - open/close 4434 (time 1746145707.06 total 40.01 last 112.15) - open/close 5630 (time 1746145717.06 total 50.01 last 119.57) - open/close 6734 (time 1746145727.06 total 60.01 last 110.39) - open/close 7860 (time 1746145737.07 total 70.01 last 112.56) - open/close 9011 (time 1746145747.07 total 80.02 last 115.04) total: 10000 open/close in 89.09 seconds: 112.24 ops/second - unlinked 0 (time 1746145758 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1218 (time 1746145824.40 total 10.00 last 121.78) - open/close 2670 (time 1746145834.40 total 20.00 last 145.17) - open/close 3821 (time 1746145844.41 total 30.01 last 115.08) - open/close 4948 (time 1746145854.42 total 40.02 last 112.56) - open/close 6030 (time 1746145864.43 total 50.03 last 108.10) - open/close 7028 (time 1746145874.45 total 60.04 last 99.63) - open/close 8057 (time 1746145884.46 total 70.06 last 102.71) - open/close 9196 (time 1746145894.47 total 80.07 last 113.88) total: 10000 open/close in 87.94 seconds: 113.72 ops/second - unlinked 0 (time 1746145904 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1080 (time 1746145971.29 total 10.00 last 107.95) - open/close 2323 (time 1746145981.29 total 20.01 last 124.29) - open/close 3428 (time 1746145991.32 total 30.03 last 110.23) - open/close 4775 (time 1746146001.32 total 40.03 last 134.66) - open/close 5927 (time 1746146011.32 total 50.03 last 115.19) - open/close 7185 (time 1746146021.33 total 60.04 last 125.66) - open/close 8118 (time 1746146031.33 total 70.05 last 93.28) - open/close 9286 (time 1746146041.34 total 80.05 last 116.73) total: 10000 open/close in 84.48 seconds: 118.37 ops/second - unlinked 0 (time 1746146047 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.141@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.141@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 PASS 69 (1381s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:35:50 (1746146150) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1619 sec ========== 20:36:24 (1746146184) === conf-sanity: start cleanup 20:36:25 (1746146185) === === conf-sanity: finish cleanup 20:36:26 (1746146186) ===