-----============= acceptance-small: conf-sanity ============----- Wed Oct 22 04:22:23 EDT 2025 oleg160-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg160-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server oleg160-server: oleg160-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg160-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server == conf-sanity test 45: long unlink handling in ptlrpcd == 04:23:42 (1761121422) start mds service on oleg160-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg160-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg160-server: LNET ready to unload oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 PASS 45 (149s) == conf-sanity test 69: replace an OST with the same index ========================================================== 04:26:10 (1761121570) start mds service on oleg160-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1532 (time 1761121622.37 total 10.01 last 153.08) - open/close 3644 (time 1761121632.38 total 20.01 last 211.12) - open/close 5858 (time 1761121642.38 total 30.01 last 221.37) - open/close 7804 (time 1761121652.38 total 40.01 last 194.58) total: 10000 open/close in 49.36 seconds: 202.60 ops/second - unlinked 0 (time 1761121663 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2623 (time 1761121697.42 total 10.00 last 262.29) - open/close 5539 (time 1761121707.43 total 20.00 last 291.53) - open/close 8144 (time 1761121717.43 total 30.01 last 260.43) total: 10000 open/close in 37.23 seconds: 268.59 ops/second - unlinked 0 (time 1761121726 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2266 (time 1761121759.85 total 10.00 last 226.55) - open/close 5164 (time 1761121769.85 total 20.00 last 289.73) - open/close 8337 (time 1761121779.85 total 30.01 last 317.24) total: 10000 open/close in 35.59 seconds: 280.98 ops/second - unlinked 0 (time 1761121787 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2589 (time 1761121823.27 total 10.00 last 258.87) - open/close 4975 (time 1761121833.27 total 20.00 last 238.59) - open/close 7547 (time 1761121843.27 total 30.00 last 257.19) - open/close 9932 (time 1761121853.27 total 40.00 last 238.44) total: 10000 open/close in 40.30 seconds: 248.14 ops/second - unlinked 0 (time 1761121855 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 3289 (time 1761121888.59 total 10.00 last 328.88) - open/close 6067 (time 1761121898.59 total 20.00 last 277.77) - open/close 8473 (time 1761121908.60 total 30.00 last 240.55) total: 10000 open/close in 35.49 seconds: 281.76 ops/second - unlinked 0 (time 1761121915 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2578 (time 1761121950.29 total 10.01 last 257.62) - open/close 4923 (time 1761121960.29 total 20.01 last 234.44) - open/close 7436 (time 1761121970.29 total 30.01 last 251.29) total: 10000 open/close in 39.53 seconds: 252.95 ops/second - unlinked 0 (time 1761121981 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2963 (time 1761122015.56 total 10.00 last 296.18) - open/close 4954 (time 1761122025.56 total 20.01 last 199.03) - open/close 5687 (time 1761122035.57 total 30.02 last 73.24) - open/close 6542 (time 1761122045.58 total 40.03 last 85.41) - open/close 7455 (time 1761122055.59 total 50.03 last 91.26) - open/close 8176 (time 1761122065.60 total 60.04 last 72.00) - open/close 8910 (time 1761122075.61 total 70.05 last 73.34) - open/close 9575 (time 1761122085.69 total 80.14 last 65.95) total: 10000 open/close in 89.95 seconds: 111.18 ops/second - unlinked 0 (time 1761122099 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 931 (time 1761122169.35 total 10.01 last 93.05) - open/close 1867 (time 1761122179.36 total 20.02 last 93.51) - open/close 2885 (time 1761122189.37 total 30.02 last 101.76) - open/close 3821 (time 1761122199.37 total 40.02 last 93.58) - open/close 4867 (time 1761122209.38 total 50.03 last 104.53) - open/close 6063 (time 1761122219.38 total 60.03 last 119.59) - open/close 6966 (time 1761122229.38 total 70.03 last 90.25) - open/close 7938 (time 1761122239.39 total 80.04 last 97.12) - open/close 8922 (time 1761122249.40 total 90.05 last 98.36) - open/close 9856 (time 1761122259.41 total 100.06 last 93.30) total: 10000 open/close in 101.58 seconds: 98.44 ops/second - unlinked 0 (time 1761122263 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 913 (time 1761122329.88 total 10.01 last 91.23) - open/close 1580 (time 1761122339.89 total 20.02 last 66.63) - open/close 2082 (time 1761122349.89 total 30.02 last 50.20) - open/close 2665 (time 1761122359.92 total 40.05 last 58.15) - open/close 3233 (time 1761122369.93 total 50.05 last 56.76) - open/close 3699 (time 1761122379.93 total 60.06 last 46.57) - open/close 4290 (time 1761122389.95 total 70.07 last 59.03) - open/close 4956 (time 1761122399.96 total 80.08 last 66.54) - open/close 5621 (time 1761122409.97 total 90.09 last 66.42) - open/close 6336 (time 1761122419.98 total 100.10 last 71.43) - open/close 7053 (time 1761122429.98 total 110.11 last 71.65) - open/close 7811 (time 1761122439.99 total 120.11 last 75.78) - open/close 8600 (time 1761122450.00 total 130.12 last 78.81) - open/close 9389 (time 1761122460.00 total 140.13 last 78.87) total: 10000 open/close in 148.45 seconds: 67.36 ops/second - unlinked 0 (time 1761122472 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 769 (time 1761122572.53 total 10.00 last 76.90) - open/close 1578 (time 1761122582.54 total 20.01 last 80.80) - open/close 2317 (time 1761122592.54 total 30.02 last 73.88) - open/close 3026 (time 1761122602.55 total 40.02 last 70.88) - open/close 3779 (time 1761122612.56 total 50.03 last 75.23) - open/close 4529 (time 1761122622.57 total 60.04 last 74.93) - open/close 5224 (time 1761122632.57 total 70.04 last 69.46) - open/close 5889 (time 1761122642.58 total 80.05 last 66.47) - open/close 6624 (time 1761122652.58 total 90.05 last 73.49) - open/close 7398 (time 1761122662.58 total 100.05 last 77.37) - open/close 8163 (time 1761122672.58 total 110.05 last 76.48) - open/close 8848 (time 1761122682.59 total 120.06 last 68.44) - open/close 9548 (time 1761122692.60 total 130.07 last 69.93) total: 10000 open/close in 136.85 seconds: 73.07 ops/second - unlinked 0 (time 1761122703 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg160-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg160-server: LNET ready to unload oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 PASS 69 (1298s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 04:47:48 (1761122868) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (4s) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 oleg160-server: oleg160-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1589 sec ========== 04:48:52 (1761122932)