-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:51:35 EDT 2025 oleg346-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg346-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg346-server oleg346-server: oleg346-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server == conf-sanity test 45: long unlink handling in ptlrpcd == 08:53:17 (1761310397) start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg346-server: LNET ready to unload oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 PASS 45 (148s) == conf-sanity test 69: replace an OST with the same index ========================================================== 08:55:45 (1761310545) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 970 (time 1761310599.40 total 10.00 last 96.96) - open/close 2026 (time 1761310609.41 total 20.01 last 105.52) - open/close 3130 (time 1761310619.42 total 30.02 last 110.32) - open/close 4725 (time 1761310629.42 total 40.02 last 159.41) - open/close 6127 (time 1761310639.43 total 50.03 last 140.14) - open/close 7430 (time 1761310649.43 total 60.03 last 130.25) - open/close 9257 (time 1761310659.43 total 70.03 last 182.67) total: 10000 open/close in 72.81 seconds: 137.34 ops/second - unlinked 0 (time 1761310663 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2217 (time 1761310698.22 total 10.00 last 221.63) - open/close 4329 (time 1761310708.23 total 20.01 last 211.10) - open/close 6573 (time 1761310718.23 total 30.01 last 224.34) - open/close 8159 (time 1761310728.23 total 40.01 last 158.60) - open/close 9898 (time 1761310738.23 total 50.01 last 173.86) total: 10000 open/close in 50.46 seconds: 198.17 ops/second - unlinked 0 (time 1761310740 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2155 (time 1761310776.02 total 10.00 last 215.50) - open/close 4183 (time 1761310786.02 total 20.00 last 202.76) - open/close 6063 (time 1761310796.02 total 30.00 last 188.00) - open/close 6844 (time 1761310806.04 total 40.02 last 77.96) - open/close 7507 (time 1761310816.05 total 50.02 last 66.28) - open/close 8175 (time 1761310826.05 total 60.03 last 66.77) - open/close 8822 (time 1761310836.06 total 70.04 last 64.64) - open/close 9523 (time 1761310846.06 total 80.04 last 70.08) total: 10000 open/close in 87.90 seconds: 113.77 ops/second - unlinked 0 (time 1761310857 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 889 (time 1761310936.64 total 10.00 last 88.88) - open/close 1761 (time 1761310946.64 total 20.01 last 87.16) - open/close 2810 (time 1761310956.65 total 30.01 last 104.88) - open/close 3796 (time 1761310966.65 total 40.01 last 98.57) - open/close 4640 (time 1761310976.65 total 50.02 last 84.36) - open/close 5356 (time 1761310986.66 total 60.03 last 71.53) - open/close 6035 (time 1761310996.67 total 70.03 last 67.88) - open/close 6778 (time 1761311006.67 total 80.03 last 74.29) - open/close 7597 (time 1761311016.68 total 90.04 last 81.82) - open/close 8397 (time 1761311026.68 total 100.04 last 79.96) - open/close 9190 (time 1761311036.68 total 110.05 last 79.28) total: 10000 open/close in 119.58 seconds: 83.63 ops/second - unlinked 0 (time 1761311048 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1329 (time 1761311114.15 total 10.00 last 132.86) - open/close 2812 (time 1761311124.15 total 20.01 last 148.23) - open/close 4439 (time 1761311134.15 total 30.01 last 162.63) - open/close 6157 (time 1761311144.15 total 40.01 last 171.79) - open/close 7673 (time 1761311154.16 total 50.02 last 151.52) - open/close 9178 (time 1761311164.16 total 60.02 last 150.49) total: 10000 open/close in 66.62 seconds: 150.10 ops/second - unlinked 0 (time 1761311172 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1479 (time 1761311220.35 total 10.01 last 147.73) - open/close 2699 (time 1761311230.35 total 20.02 last 121.93) - open/close 3856 (time 1761311240.36 total 30.02 last 115.64) - open/close 4911 (time 1761311250.36 total 40.03 last 105.46) - open/close 6092 (time 1761311260.36 total 50.03 last 118.08) - open/close 7480 (time 1761311270.37 total 60.03 last 138.71) - open/close 9001 (time 1761311280.38 total 70.04 last 152.03) total: 10000 open/close in 80.00 seconds: 124.99 ops/second - unlinked 0 (time 1761311295 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 877 (time 1761311377.46 total 10.01 last 87.63) - open/close 1785 (time 1761311387.46 total 20.01 last 90.78) - open/close 2655 (time 1761311397.47 total 30.02 last 86.95) - open/close 3199 (time 1761311407.48 total 40.03 last 54.35) - open/close 3756 (time 1761311417.48 total 50.03 last 55.68) - open/close 4304 (time 1761311427.50 total 60.05 last 54.71) - open/close 4930 (time 1761311437.50 total 70.05 last 62.56) - open/close 5555 (time 1761311447.51 total 80.06 last 62.44) - open/close 6307 (time 1761311457.52 total 90.07 last 75.15) - open/close 7031 (time 1761311467.53 total 100.08 last 72.29) - open/close 7822 (time 1761311477.54 total 110.09 last 79.03) - open/close 8654 (time 1761311487.55 total 120.10 last 83.14) - open/close 9421 (time 1761311497.55 total 130.11 last 76.67) total: 10000 open/close in 136.83 seconds: 73.08 ops/second - unlinked 0 (time 1761311508 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 788 (time 1761311596.29 total 10.01 last 78.74) - open/close 1575 (time 1761311606.30 total 20.01 last 78.67) - open/close 2388 (time 1761311616.32 total 30.03 last 81.14) - open/close 3263 (time 1761311626.33 total 40.04 last 87.41) - open/close 4102 (time 1761311636.34 total 50.06 last 83.79) - open/close 4953 (time 1761311646.34 total 60.06 last 85.09) - open/close 5788 (time 1761311656.35 total 70.06 last 83.47) - open/close 6595 (time 1761311666.36 total 80.07 last 80.61) - open/close 7423 (time 1761311676.37 total 90.08 last 82.72) - open/close 8232 (time 1761311686.39 total 100.10 last 80.75) - open/close 8972 (time 1761311696.39 total 110.10 last 73.97) - open/close 9733 (time 1761311706.40 total 120.11 last 76.04) total: 10000 open/close in 123.88 seconds: 80.73 ops/second - unlinked 0 (time 1761311713 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 895 (time 1761311803.50 total 10.00 last 89.47) - open/close 1735 (time 1761311813.51 total 20.01 last 83.94) - open/close 2599 (time 1761311823.51 total 30.01 last 86.40) - open/close 3465 (time 1761311833.51 total 40.01 last 86.58) - open/close 4338 (time 1761311843.52 total 50.02 last 87.27) - open/close 5209 (time 1761311853.53 total 60.03 last 87.02) - open/close 6053 (time 1761311863.53 total 70.03 last 84.34) - open/close 6743 (time 1761311873.54 total 80.04 last 68.93) - open/close 7632 (time 1761311883.55 total 90.04 last 88.90) - open/close 8490 (time 1761311893.56 total 100.06 last 85.70) - open/close 9353 (time 1761311903.56 total 110.06 last 86.25) total: 10000 open/close in 117.53 seconds: 85.09 ops/second - unlinked 0 (time 1761311914 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 871 (time 1761311997.88 total 10.01 last 87.05) - open/close 1755 (time 1761312007.88 total 20.01 last 88.38) - open/close 2628 (time 1761312017.89 total 30.01 last 87.24) - open/close 3447 (time 1761312027.89 total 40.02 last 81.89) - open/close 4297 (time 1761312037.90 total 50.02 last 84.92) - open/close 5170 (time 1761312047.91 total 60.04 last 87.17) - open/close 6000 (time 1761312057.92 total 70.05 last 82.96) - open/close 6853 (time 1761312067.93 total 80.05 last 85.22) - open/close 7720 (time 1761312077.93 total 90.06 last 86.66) - open/close 8606 (time 1761312087.94 total 100.07 last 88.54) - open/close 9473 (time 1761312097.94 total 110.07 last 86.69) total: 10000 open/close in 116.09 seconds: 86.14 ops/second - unlinked 0 (time 1761312106 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg346-server: LNET ready to unload oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 PASS 69 (1709s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 09:24:14 (1761312254) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (4s) Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: oleg346-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2015 sec ========== 09:25:10 (1761312310)