-----============= acceptance-small: conf-sanity ============----- Thu May 1 23:04:36 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg237-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg237-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server oleg237-server: oleg237-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server == conf-sanity test 45: long unlink handling in ptlrpcd == 23:06:08 (1746155168) start mds service on oleg237-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg237-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 PASS 45 (147s) == conf-sanity test 69: replace an OST with the same index ========================================================== 23:08:36 (1746155316) start mds service on oleg237-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1313 (time 1746155359.60 total 10.00 last 131.29) - open/close 2593 (time 1746155369.60 total 20.00 last 127.97) - open/close 3776 (time 1746155379.60 total 30.00 last 118.29) - open/close 5001 (time 1746155389.61 total 40.01 last 122.46) - open/close 6001 (time 1746155399.61 total 50.01 last 99.95) - open/close 7205 (time 1746155409.62 total 60.02 last 120.28) - open/close 8366 (time 1746155419.62 total 70.02 last 116.10) - open/close 9484 (time 1746155429.63 total 80.03 last 111.71) total: 10000 open/close in 84.72 seconds: 118.03 ops/second - unlinked 0 (time 1746155436 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1121 (time 1746155500.94 total 10.01 last 111.99) - open/close 2307 (time 1746155510.94 total 20.01 last 118.54) - open/close 3466 (time 1746155520.95 total 30.02 last 115.85) - open/close 4696 (time 1746155530.95 total 40.02 last 122.96) - open/close 5903 (time 1746155540.95 total 50.02 last 120.69) - open/close 6920 (time 1746155550.96 total 60.03 last 101.64) - open/close 8086 (time 1746155560.96 total 70.03 last 116.55) - open/close 9023 (time 1746155570.97 total 80.04 last 93.61) total: 10000 open/close in 89.57 seconds: 111.64 ops/second - unlinked 0 (time 1746155582 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1450 (time 1746155640.35 total 10.00 last 144.96) - open/close 2837 (time 1746155650.36 total 20.01 last 138.57) - open/close 4064 (time 1746155660.36 total 30.02 last 122.64) - open/close 5271 (time 1746155670.36 total 40.02 last 120.65) - open/close 6428 (time 1746155680.37 total 50.03 last 115.63) - open/close 7968 (time 1746155690.38 total 60.03 last 153.90) - open/close 9028 (time 1746155700.38 total 70.04 last 105.98) total: 10000 open/close in 79.46 seconds: 125.86 ops/second - unlinked 0 (time 1746155711 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1301 (time 1746155774.97 total 10.00 last 130.10) - open/close 2542 (time 1746155784.97 total 20.00 last 124.10) - open/close 4093 (time 1746155794.97 total 30.00 last 155.09) - open/close 5532 (time 1746155804.97 total 40.00 last 143.90) - open/close 7104 (time 1746155814.98 total 50.01 last 157.02) - open/close 8542 (time 1746155824.98 total 60.02 last 143.73) - open/close 9629 (time 1746155834.99 total 70.02 last 108.67) total: 10000 open/close in 73.69 seconds: 135.71 ops/second - unlinked 0 (time 1746155840 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1246 (time 1746155908.30 total 10.00 last 124.59) - open/close 2374 (time 1746155918.30 total 20.00 last 112.79) - open/close 3377 (time 1746155928.32 total 30.02 last 100.16) - open/close 4374 (time 1746155938.32 total 40.02 last 99.69) - open/close 5310 (time 1746155948.32 total 50.02 last 93.60) - open/close 6240 (time 1746155958.33 total 60.02 last 92.94) - open/close 7194 (time 1746155968.33 total 70.03 last 95.35) - open/close 8048 (time 1746155978.34 total 80.04 last 85.29) - open/close 8968 (time 1746155988.35 total 90.04 last 91.97) total: 10000 open/close in 100.04 seconds: 99.96 ops/second - unlinked 0 (time 1746156000 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1273 (time 1746156065.73 total 10.01 last 127.23) - open/close 2478 (time 1746156075.73 total 20.01 last 120.48) - open/close 3674 (time 1746156085.74 total 30.01 last 119.54) - open/close 4972 (time 1746156095.74 total 40.02 last 129.75) - open/close 6477 (time 1746156105.74 total 50.02 last 150.44) - open/close 7632 (time 1746156115.75 total 60.02 last 115.47) - open/close 9091 (time 1746156125.75 total 70.03 last 145.78) total: 10000 open/close in 76.94 seconds: 129.98 ops/second - unlinked 0 (time 1746156134 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1352 (time 1746156190.90 total 10.00 last 135.16) - open/close 2446 (time 1746156200.91 total 20.01 last 109.31) - open/close 3636 (time 1746156210.91 total 30.01 last 118.97) - open/close 4952 (time 1746156220.92 total 40.02 last 131.54) - open/close 6072 (time 1746156230.92 total 50.02 last 111.95) - open/close 7291 (time 1746156240.93 total 60.03 last 121.82) - open/close 8467 (time 1746156250.93 total 70.04 last 117.52) - open/close 9609 (time 1746156260.94 total 80.04 last 114.11) total: 10000 open/close in 83.00 seconds: 120.49 ops/second - unlinked 0 (time 1746156265 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1302 (time 1746156325.45 total 10.00 last 130.18) - open/close 2485 (time 1746156335.45 total 20.01 last 118.24) - open/close 3794 (time 1746156345.46 total 30.01 last 130.85) - open/close 4923 (time 1746156355.48 total 40.03 last 112.70) - open/close 6138 (time 1746156365.49 total 50.04 last 121.39) - open/close 7359 (time 1746156375.49 total 60.04 last 122.04) - open/close 8332 (time 1746156385.50 total 70.05 last 97.23) - open/close 9392 (time 1746156395.50 total 80.05 last 106.00) total: 10000 open/close in 86.70 seconds: 115.35 ops/second - unlinked 0 (time 1746156404 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1079 (time 1746156468.96 total 10.00 last 107.86) - open/close 2165 (time 1746156478.96 total 20.00 last 108.58) - open/close 3322 (time 1746156488.97 total 30.01 last 115.62) - open/close 4534 (time 1746156498.97 total 40.02 last 121.11) - open/close 5696 (time 1746156508.98 total 50.02 last 116.16) - open/close 6800 (time 1746156518.99 total 60.03 last 110.31) - open/close 7853 (time 1746156529.00 total 70.05 last 105.12) - open/close 8797 (time 1746156539.02 total 80.06 last 94.28) - open/close 9782 (time 1746156549.02 total 90.07 last 98.43) total: 10000 open/close in 92.31 seconds: 108.33 ops/second - unlinked 0 (time 1746156553 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1823 (time 1746156610.62 total 10.01 last 182.20) - open/close 3430 (time 1746156620.62 total 20.01 last 160.66) - open/close 4958 (time 1746156630.63 total 30.02 last 152.66) - open/close 6705 (time 1746156640.63 total 40.02 last 174.68) - open/close 8075 (time 1746156650.63 total 50.02 last 136.95) - open/close 9558 (time 1746156660.63 total 60.02 last 148.28) total: 10000 open/close in 62.90 seconds: 158.98 ops/second - unlinked 0 (time 1746156665 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 PASS 69 (1436s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 23:32:32 (1746156752) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 oleg237-server: oleg237-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1704 sec ========== 23:33:01 (1746156781) === conf-sanity: start cleanup 23:33:02 (1746156782) === === conf-sanity: finish cleanup 23:33:03 (1746156783) ===