-----============= acceptance-small: conf-sanity ============----- Fri Jul 25 06:44:59 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg156-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg156-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg156-server oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg156-server: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg156-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server == conf-sanity test 45: long unlink handling in ptlrpcd == 06:46:43 (1753440403) start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 PASS 45 (144s) == conf-sanity test 69: replace an OST with the same index ========================================================== 06:49:07 (1753440547) start mds service on oleg156-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1361 (time 1753440593.43 total 10.00 last 136.07) - open/close 2422 (time 1753440603.43 total 20.01 last 106.06) - open/close 3417 (time 1753440613.44 total 30.01 last 99.43) - open/close 4401 (time 1753440623.44 total 40.02 last 98.32) - open/close 5548 (time 1753440633.45 total 50.02 last 114.70) - open/close 6896 (time 1753440643.45 total 60.02 last 134.77) - open/close 8003 (time 1753440653.45 total 70.03 last 110.66) - open/close 9211 (time 1753440663.45 total 80.03 last 120.77) total: 10000 open/close in 88.02 seconds: 113.61 ops/second - unlinked 0 (time 1753440673 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1252 (time 1753440723.50 total 10.00 last 125.16) - open/close 2702 (time 1753440733.51 total 20.01 last 144.86) - open/close 4121 (time 1753440743.52 total 30.01 last 141.87) - open/close 5661 (time 1753440753.52 total 40.02 last 153.90) - open/close 7534 (time 1753440763.53 total 50.02 last 187.25) - open/close 9502 (time 1753440773.53 total 60.03 last 196.64) total: 10000 open/close in 62.75 seconds: 159.37 ops/second - unlinked 0 (time 1753440778 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1916 (time 1753440819.98 total 10.00 last 191.57) - open/close 3681 (time 1753440829.99 total 20.01 last 176.42) - open/close 5167 (time 1753440840.00 total 30.02 last 148.44) - open/close 6780 (time 1753440850.00 total 40.02 last 161.25) - open/close 8662 (time 1753440860.00 total 50.02 last 188.15) total: 10000 open/close in 55.42 seconds: 180.44 ops/second - unlinked 0 (time 1753440866 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1655 (time 1753440908.87 total 10.00 last 165.46) - open/close 3210 (time 1753440918.87 total 20.00 last 155.49) - open/close 4955 (time 1753440928.88 total 30.01 last 174.43) - open/close 6313 (time 1753440938.88 total 40.02 last 135.68) - open/close 7740 (time 1753440948.89 total 50.02 last 142.66) - open/close 9252 (time 1753440958.89 total 60.02 last 151.19) total: 10000 open/close in 64.24 seconds: 155.66 ops/second - unlinked 0 (time 1753440964 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 503 (time 1753441021.91 total 10.02 last 50.22) - open/close 901 (time 1753441031.92 total 20.03 last 39.73) - open/close 1236 (time 1753441042.00 total 30.11 last 33.24) - open/close 1783 (time 1753441052.01 total 40.12 last 54.64) - open/close 2374 (time 1753441062.02 total 50.13 last 59.05) - open/close 2936 (time 1753441072.04 total 60.15 last 56.11) - open/close 3619 (time 1753441082.06 total 70.17 last 68.15) - open/close 4337 (time 1753441092.07 total 80.18 last 71.74) - open/close 5049 (time 1753441102.08 total 90.18 last 71.15) - open/close 5749 (time 1753441112.08 total 100.19 last 69.94) - open/close 6444 (time 1753441122.10 total 110.20 last 69.41) - open/close 7165 (time 1753441132.10 total 120.21 last 72.05) - open/close 7901 (time 1753441142.11 total 130.22 last 73.52) - open/close 8643 (time 1753441152.13 total 140.24 last 74.07) - open/close 9354 (time 1753441162.15 total 150.26 last 70.99) total: 10000 open/close in 159.44 seconds: 62.72 ops/second - unlinked 0 (time 1753441175 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 666 (time 1753441271.01 total 10.01 last 66.53) - open/close 1288 (time 1753441281.02 total 20.02 last 62.14) - open/close 1945 (time 1753441291.03 total 30.02 last 65.67) - open/close 2628 (time 1753441301.04 total 40.04 last 68.21) - open/close 3267 (time 1753441311.04 total 50.04 last 63.86) - open/close 3940 (time 1753441321.05 total 60.05 last 67.24) - open/close 4626 (time 1753441331.06 total 70.05 last 68.58) - open/close 5320 (time 1753441341.06 total 80.06 last 69.35) - open/close 6029 (time 1753441351.07 total 90.07 last 70.85) - open/close 6750 (time 1753441361.09 total 100.09 last 71.96) - open/close 7468 (time 1753441371.09 total 110.09 last 71.77) - open/close 8175 (time 1753441381.10 total 120.10 last 70.68) - open/close 8871 (time 1753441391.10 total 130.10 last 69.55) - open/close 9557 (time 1753441401.11 total 140.11 last 68.56) total: 10000 open/close in 146.39 seconds: 68.31 ops/second - unlinked 0 (time 1753441411 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 806 (time 1753441493.69 total 10.01 last 80.53) - open/close 1774 (time 1753441503.70 total 20.02 last 96.73) - open/close 2847 (time 1753441513.71 total 30.02 last 107.23) - open/close 3871 (time 1753441523.71 total 40.03 last 102.36) - open/close 4696 (time 1753441533.71 total 50.03 last 82.49) - open/close 5582 (time 1753441543.71 total 60.03 last 88.59) - open/close 6601 (time 1753441553.72 total 70.03 last 101.86) - open/close 7496 (time 1753441563.73 total 80.05 last 89.36) - open/close 8507 (time 1753441573.75 total 90.07 last 100.92) - open/close 9448 (time 1753441583.76 total 100.08 last 93.98) total: 10000 open/close in 106.04 seconds: 94.30 ops/second - unlinked 0 (time 1753441592 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1427 (time 1753441649.40 total 10.00 last 142.68) - open/close 2664 (time 1753441659.41 total 20.01 last 123.63) - open/close 3762 (time 1753441669.42 total 30.02 last 109.70) - open/close 4834 (time 1753441679.42 total 40.02 last 107.20) - open/close 6105 (time 1753441689.42 total 50.02 last 127.07) - open/close 7514 (time 1753441699.43 total 60.03 last 140.68) - open/close 8855 (time 1753441709.44 total 70.04 last 134.05) total: 10000 open/close in 78.64 seconds: 127.16 ops/second - unlinked 0 (time 1753441720 ; total 1 ; last 1) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1447 (time 1753441774.84 total 10.00 last 144.69) - open/close 2897 (time 1753441784.84 total 20.00 last 144.99) - open/close 4132 (time 1753441794.85 total 30.01 last 123.39) - open/close 5131 (time 1753441804.85 total 40.02 last 99.85) - open/close 6293 (time 1753441814.85 total 50.02 last 116.17) - open/close 7771 (time 1753441824.86 total 60.02 last 147.75) - open/close 9169 (time 1753441834.86 total 70.03 last 139.69) total: 10000 open/close in 76.09 seconds: 131.43 ops/second - unlinked 0 (time 1753441842 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1734 (time 1753441895.26 total 10.01 last 173.18) - open/close 3441 (time 1753441905.26 total 20.02 last 170.64) - open/close 5103 (time 1753441915.27 total 30.03 last 166.04) - open/close 6810 (time 1753441925.27 total 40.03 last 170.66) - open/close 8841 (time 1753441935.27 total 50.03 last 203.09) total: 10000 open/close in 56.40 seconds: 177.29 ops/second - unlinked 0 (time 1753441943 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.156@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.156@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg156-server: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg156-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre On OST0, 18725 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 PASS 69 (1494s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 07:14:02 (1753442042) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1780 sec ========== 07:14:40 (1753442080) === conf-sanity: start cleanup 07:14:41 (1753442081) === === conf-sanity: finish cleanup 07:14:43 (1753442083) ===