-----============= acceptance-small: conf-sanity ============----- Thu Jul 24 09:42:54 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg201-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg201-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg201-server oleg201-server: oleg201-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server == conf-sanity test 45: long unlink handling in ptlrpcd == 09:44:29 (1753364669) start mds service on oleg201-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg201-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 PASS 45 (139s) == conf-sanity test 69: replace an OST with the same index ========================================================== 09:46:48 (1753364808) start mds service on oleg201-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2021 (time 1753364852.11 total 10.00 last 202.10) - open/close 4083 (time 1753364862.11 total 20.01 last 206.10) - open/close 6521 (time 1753364872.12 total 30.01 last 243.69) - open/close 8679 (time 1753364882.12 total 40.01 last 215.69) total: 10000 open/close in 49.17 seconds: 203.38 ops/second - unlinked 0 (time 1753364892 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2196 (time 1753364929.86 total 10.00 last 219.58) - open/close 4552 (time 1753364939.86 total 20.00 last 235.59) - open/close 5946 (time 1753364949.87 total 30.01 last 139.28) - open/close 6764 (time 1753364959.88 total 40.02 last 81.73) - open/close 7580 (time 1753364969.88 total 50.02 last 81.59) - open/close 8440 (time 1753364979.88 total 60.02 last 85.99) - open/close 9563 (time 1753364989.89 total 70.03 last 112.18) total: 10000 open/close in 73.91 seconds: 135.30 ops/second - unlinked 0 (time 1753364995 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1352 (time 1753365043.47 total 10.01 last 135.02) - open/close 2691 (time 1753365053.47 total 20.02 last 133.83) - open/close 4241 (time 1753365063.47 total 30.02 last 155.00) - open/close 5575 (time 1753365073.47 total 40.02 last 133.38) - open/close 7116 (time 1753365083.48 total 50.02 last 154.07) - open/close 8470 (time 1753365093.48 total 60.03 last 135.34) - open/close 9748 (time 1753365103.48 total 70.03 last 127.77) total: 10000 open/close in 72.01 seconds: 138.87 ops/second - unlinked 0 (time 1753365107 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2198 (time 1753365150.84 total 10.00 last 219.74) - open/close 3906 (time 1753365160.84 total 20.01 last 170.75) - open/close 5444 (time 1753365170.84 total 30.01 last 153.79) - open/close 7177 (time 1753365180.84 total 40.01 last 173.29) - open/close 8969 (time 1753365190.85 total 50.02 last 179.01) total: 10000 open/close in 55.34 seconds: 180.70 ops/second - unlinked 0 (time 1753365197 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1927 (time 1753365230.96 total 10.00 last 192.65) - open/close 4022 (time 1753365240.97 total 20.01 last 209.40) - open/close 5997 (time 1753365250.97 total 30.01 last 197.44) - open/close 8108 (time 1753365260.97 total 40.01 last 211.10) total: 10000 open/close in 49.53 seconds: 201.89 ops/second - unlinked 0 (time 1753365272 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2070 (time 1753365307.56 total 10.00 last 206.99) - open/close 4482 (time 1753365317.56 total 20.01 last 241.03) - open/close 6760 (time 1753365327.57 total 30.01 last 227.77) - open/close 9057 (time 1753365337.57 total 40.01 last 229.69) total: 10000 open/close in 43.95 seconds: 227.54 ops/second - unlinked 0 (time 1753365343 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2696 (time 1753365372.10 total 10.00 last 269.57) - open/close 5309 (time 1753365382.11 total 20.00 last 261.22) - open/close 8274 (time 1753365392.11 total 30.01 last 296.41) total: 10000 open/close in 36.48 seconds: 274.13 ops/second - unlinked 0 (time 1753365400 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3556 (time 1753365432.59 total 10.00 last 355.50) - open/close 6940 (time 1753365442.59 total 20.01 last 338.31) - open/close 9634 (time 1753365452.60 total 30.01 last 269.32) total: 10000 open/close in 31.48 seconds: 317.66 ops/second - unlinked 0 (time 1753365455 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2749 (time 1753365488.10 total 10.00 last 274.87) - open/close 4989 (time 1753365498.10 total 20.00 last 223.93) - open/close 7530 (time 1753365508.10 total 30.01 last 254.02) total: 10000 open/close in 38.11 seconds: 262.37 ops/second - unlinked 0 (time 1753365517 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2697 (time 1753365547.19 total 10.00 last 269.65) - open/close 5583 (time 1753365557.19 total 20.00 last 288.60) - open/close 8865 (time 1753365567.19 total 30.00 last 328.16) total: 10000 open/close in 33.26 seconds: 300.63 ops/second - unlinked 0 (time 1753365571 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.101@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.101@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre On OST0, 18757 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 PASS 69 (825s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 10:00:34 (1753365634) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1086 sec ========== 10:01:01 (1753365661) === conf-sanity: start cleanup 10:01:01 (1753365661) === === conf-sanity: finish cleanup 10:01:02 (1753365662) ===