-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:19:35 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg250-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg250-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:21:04 (1746138064) start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 PASS 45 (144s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:23:28 (1746138208) start mds service on oleg250-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1045 (time 1746138255.35 total 10.01 last 104.41) - open/close 2178 (time 1746138265.35 total 20.01 last 113.29) - open/close 3223 (time 1746138275.38 total 30.03 last 104.24) - open/close 4292 (time 1746138285.39 total 40.04 last 106.83) - open/close 5350 (time 1746138295.40 total 50.05 last 105.68) - open/close 6619 (time 1746138305.40 total 60.06 last 126.83) - open/close 7771 (time 1746138315.41 total 70.07 last 115.09) - open/close 8853 (time 1746138325.43 total 80.08 last 108.06) - open/close 9947 (time 1746138335.43 total 90.08 last 109.38) total: 10000 open/close in 90.62 seconds: 110.35 ops/second - unlinked 0 (time 1746138337 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1464 (time 1746138396.96 total 10.00 last 146.37) - open/close 2613 (time 1746138406.96 total 20.00 last 114.88) - open/close 3691 (time 1746138416.96 total 30.01 last 107.73) - open/close 4804 (time 1746138426.97 total 40.01 last 111.26) - open/close 5800 (time 1746138436.97 total 50.02 last 99.58) - open/close 6941 (time 1746138446.97 total 60.02 last 114.10) - open/close 8143 (time 1746138456.98 total 70.02 last 120.12) - open/close 9279 (time 1746138466.99 total 80.03 last 113.49) total: 10000 open/close in 86.26 seconds: 115.94 ops/second - unlinked 0 (time 1746138474 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1167 (time 1746138532.83 total 10.01 last 116.63) - open/close 2481 (time 1746138542.84 total 20.01 last 131.32) - open/close 3823 (time 1746138552.84 total 30.01 last 134.20) - open/close 5311 (time 1746138562.84 total 40.02 last 148.69) - open/close 6570 (time 1746138572.85 total 50.02 last 125.87) - open/close 8106 (time 1746138582.85 total 60.02 last 153.58) - open/close 9347 (time 1746138592.85 total 70.03 last 124.03) total: 10000 open/close in 75.10 seconds: 133.15 ops/second - unlinked 0 (time 1746138599 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1068 (time 1746138653.09 total 10.00 last 106.75) - open/close 2396 (time 1746138663.09 total 20.01 last 132.72) - open/close 3549 (time 1746138673.10 total 30.02 last 115.23) - open/close 4786 (time 1746138683.10 total 40.02 last 123.70) - open/close 6273 (time 1746138693.10 total 50.02 last 148.64) - open/close 8106 (time 1746138703.10 total 60.02 last 183.28) - open/close 9781 (time 1746138713.11 total 70.03 last 167.45) total: 10000 open/close in 71.95 seconds: 138.98 ops/second - unlinked 0 (time 1746138716 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1129 (time 1746138777.96 total 10.00 last 112.89) - open/close 2576 (time 1746138787.96 total 20.00 last 144.64) - open/close 3969 (time 1746138797.96 total 30.01 last 139.26) - open/close 5210 (time 1746138807.97 total 40.01 last 124.06) - open/close 6524 (time 1746138817.98 total 50.02 last 131.26) - open/close 7609 (time 1746138827.98 total 60.02 last 108.49) - open/close 8682 (time 1746138837.99 total 70.03 last 107.18) - open/close 9935 (time 1746138848.00 total 80.04 last 125.24) total: 10000 open/close in 80.55 seconds: 124.15 ops/second - unlinked 0 (time 1746138850 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1473 (time 1746138907.08 total 10.00 last 147.23) - open/close 3108 (time 1746138917.08 total 20.01 last 163.44) - open/close 4527 (time 1746138927.08 total 30.01 last 141.90) - open/close 5810 (time 1746138937.08 total 40.01 last 128.27) - open/close 6930 (time 1746138947.08 total 50.01 last 111.97) - open/close 8151 (time 1746138957.09 total 60.02 last 122.08) - open/close 9290 (time 1746138967.09 total 70.02 last 113.83) total: 10000 open/close in 75.93 seconds: 131.69 ops/second - unlinked 0 (time 1746138974 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1267 (time 1746139032.44 total 10.00 last 126.67) - open/close 2482 (time 1746139042.46 total 20.01 last 121.37) - open/close 3830 (time 1746139052.46 total 30.02 last 134.75) - open/close 5716 (time 1746139062.46 total 40.02 last 188.57) - open/close 7522 (time 1746139072.46 total 50.02 last 180.57) - open/close 9250 (time 1746139082.46 total 60.02 last 172.77) total: 10000 open/close in 64.37 seconds: 155.36 ops/second - unlinked 0 (time 1746139088 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 2071 (time 1746139140.35 total 10.00 last 207.09) - open/close 3840 (time 1746139150.35 total 20.00 last 176.89) - open/close 5618 (time 1746139160.35 total 30.00 last 177.75) - open/close 7393 (time 1746139170.35 total 40.00 last 177.48) - open/close 8914 (time 1746139180.35 total 50.01 last 152.08) total: 10000 open/close in 56.69 seconds: 176.41 ops/second - unlinked 0 (time 1746139188 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1445 (time 1746139236.09 total 10.00 last 144.45) - open/close 3073 (time 1746139246.09 total 20.01 last 162.76) - open/close 4356 (time 1746139256.09 total 30.01 last 128.28) - open/close 5845 (time 1746139266.10 total 40.01 last 148.87) - open/close 7518 (time 1746139276.10 total 50.01 last 167.22) - open/close 8812 (time 1746139286.10 total 60.02 last 129.40) total: 10000 open/close in 67.33 seconds: 148.53 ops/second - unlinked 0 (time 1746139294 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1441 (time 1746139343.54 total 10.00 last 144.05) - open/close 2900 (time 1746139353.54 total 20.01 last 145.86) - open/close 4253 (time 1746139363.55 total 30.01 last 135.23) - open/close 5600 (time 1746139373.56 total 40.02 last 134.57) - open/close 6755 (time 1746139383.56 total 50.02 last 115.46) - open/close 8089 (time 1746139393.56 total 60.03 last 133.38) - open/close 9624 (time 1746139403.57 total 70.03 last 153.44) total: 10000 open/close in 72.98 seconds: 137.03 ops/second - unlinked 0 (time 1746139408 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre On OST0, 18389 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 PASS 69 (1283s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 18:44:52 (1746139492) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1553 sec ========== 18:45:28 (1746139528) === conf-sanity: start cleanup 18:45:29 (1746139529) === === conf-sanity: finish cleanup 18:45:30 (1746139530) ===