-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 11:06:31 EDT 2025 oleg114-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg114-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg114-server oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg114-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server == conf-sanity test 45: long unlink handling in ptlrpcd == 11:08:22 (1761318502) start mds service on oleg114-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg114-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg114-server: LNET ready to unload oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 PASS 45 (196s) == conf-sanity test 69: replace an OST with the same index ========================================================== 11:11:38 (1761318698) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 807 (time 1761318767.06 total 10.01 last 80.64) - open/close 1647 (time 1761318777.07 total 20.01 last 83.94) - open/close 2488 (time 1761318787.07 total 30.02 last 84.09) - open/close 3289 (time 1761318797.07 total 40.02 last 80.08) - open/close 4071 (time 1761318807.08 total 50.02 last 78.19) - open/close 4722 (time 1761318817.08 total 60.03 last 65.06) - open/close 5464 (time 1761318827.09 total 70.03 last 74.13) - open/close 6258 (time 1761318837.10 total 80.04 last 79.34) - open/close 7078 (time 1761318847.10 total 90.05 last 81.97) - open/close 7857 (time 1761318857.12 total 100.06 last 77.80) - open/close 8715 (time 1761318867.13 total 110.07 last 85.70) - open/close 9549 (time 1761318877.13 total 120.07 last 83.39) total: 10000 open/close in 125.60 seconds: 79.62 ops/second - unlinked 0 (time 1761318886 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 913 (time 1761318964.87 total 10.01 last 91.23) - open/close 1741 (time 1761318974.88 total 20.02 last 82.73) - open/close 2589 (time 1761318984.88 total 30.02 last 84.77) - open/close 3425 (time 1761318994.89 total 40.03 last 83.53) - open/close 4252 (time 1761319004.90 total 50.04 last 82.59) - open/close 5105 (time 1761319014.91 total 60.05 last 85.25) - open/close 6040 (time 1761319024.92 total 70.06 last 93.40) - open/close 7017 (time 1761319034.93 total 80.07 last 97.63) - open/close 7965 (time 1761319044.93 total 90.07 last 94.77) - open/close 8969 (time 1761319054.95 total 100.08 last 100.25) - open/close 9822 (time 1761319064.95 total 110.09 last 85.22) total: 10000 open/close in 112.27 seconds: 89.07 ops/second - unlinked 0 (time 1761319069 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1167 (time 1761319136.55 total 10.00 last 116.65) - open/close 2082 (time 1761319146.56 total 20.01 last 91.45) - open/close 2946 (time 1761319156.57 total 30.02 last 86.32) - open/close 3913 (time 1761319166.57 total 40.02 last 96.67) - open/close 4928 (time 1761319176.57 total 50.02 last 101.48) - open/close 6048 (time 1761319186.58 total 60.03 last 111.93) - open/close 7152 (time 1761319196.58 total 70.03 last 110.39) - open/close 8409 (time 1761319206.59 total 80.04 last 125.62) - open/close 9396 (time 1761319216.59 total 90.04 last 98.69) total: 10000 open/close in 96.94 seconds: 103.15 ops/second - unlinked 0 (time 1761319226 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1017 (time 1761319294.26 total 10.01 last 101.61) - open/close 2385 (time 1761319304.26 total 20.02 last 136.71) - open/close 3560 (time 1761319314.26 total 30.02 last 117.49) - open/close 4560 (time 1761319324.27 total 40.02 last 99.93) - open/close 5551 (time 1761319334.28 total 50.03 last 99.03) - open/close 6942 (time 1761319344.29 total 60.04 last 138.99) - open/close 8403 (time 1761319354.30 total 70.05 last 145.95) - open/close 9765 (time 1761319364.31 total 80.06 last 136.06) total: 10000 open/close in 82.36 seconds: 121.42 ops/second - unlinked 0 (time 1761319369 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1188 (time 1761319428.90 total 10.00 last 118.76) - open/close 2473 (time 1761319438.91 total 20.01 last 128.46) - open/close 3675 (time 1761319448.91 total 30.01 last 120.19) - open/close 4997 (time 1761319458.91 total 40.01 last 132.20) - open/close 6651 (time 1761319468.91 total 50.01 last 165.35) - open/close 8040 (time 1761319478.91 total 60.01 last 138.85) - open/close 9396 (time 1761319488.92 total 70.02 last 135.53) total: 10000 open/close in 74.04 seconds: 135.06 ops/second - unlinked 0 (time 1761319494 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1500 (time 1761319553.21 total 10.01 last 149.92) - open/close 2738 (time 1761319563.21 total 20.01 last 123.78) - open/close 3780 (time 1761319573.22 total 30.01 last 104.15) - open/close 4904 (time 1761319583.22 total 40.02 last 112.32) - open/close 6098 (time 1761319593.24 total 50.04 last 119.15) - open/close 6707 (time 1761319603.25 total 60.04 last 60.89) - open/close 7278 (time 1761319613.25 total 70.05 last 57.08) - open/close 7816 (time 1761319623.26 total 80.05 last 53.76) - open/close 8324 (time 1761319633.27 total 90.07 last 50.74) - open/close 8910 (time 1761319643.28 total 100.08 last 58.55) - open/close 9592 (time 1761319653.28 total 110.08 last 68.17) total: 10000 open/close in 115.29 seconds: 86.74 ops/second - unlinked 0 (time 1761319661 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 825 (time 1761319748.42 total 10.00 last 82.47) - open/close 1611 (time 1761319758.43 total 20.01 last 78.53) - open/close 2427 (time 1761319768.43 total 30.02 last 81.56) - open/close 3235 (time 1761319778.43 total 40.02 last 80.80) - open/close 4042 (time 1761319788.45 total 50.03 last 80.59) - open/close 4906 (time 1761319798.45 total 60.03 last 86.40) - open/close 5695 (time 1761319808.45 total 70.03 last 78.88) - open/close 6551 (time 1761319818.45 total 80.04 last 85.58) - open/close 7307 (time 1761319828.46 total 90.04 last 75.54) - open/close 8022 (time 1761319838.47 total 100.05 last 71.45) - open/close 8823 (time 1761319848.48 total 110.06 last 80.02) - open/close 9636 (time 1761319858.49 total 120.07 last 81.21) total: 10000 open/close in 124.84 seconds: 80.10 ops/second - unlinked 0 (time 1761319866 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 753 (time 1761319957.57 total 10.01 last 75.23) - open/close 1537 (time 1761319967.58 total 20.02 last 78.34) - open/close 2378 (time 1761319977.58 total 30.02 last 84.07) - open/close 3183 (time 1761319987.60 total 40.04 last 80.38) - open/close 3952 (time 1761319997.60 total 50.04 last 76.88) - open/close 4790 (time 1761320007.60 total 60.04 last 83.78) - open/close 5540 (time 1761320017.61 total 70.05 last 74.95) - open/close 6293 (time 1761320027.61 total 80.06 last 75.25) - open/close 7130 (time 1761320037.62 total 90.06 last 83.66) - open/close 7927 (time 1761320047.62 total 100.07 last 79.66) - open/close 8671 (time 1761320057.63 total 110.07 last 74.37) - open/close 9447 (time 1761320067.63 total 120.08 last 77.54) total: 10000 open/close in 127.22 seconds: 78.61 ops/second - unlinked 0 (time 1761320078 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 783 (time 1761320162.89 total 10.01 last 78.23) - open/close 1576 (time 1761320172.90 total 20.02 last 79.22) - open/close 2362 (time 1761320182.91 total 30.03 last 78.51) - open/close 3147 (time 1761320192.92 total 40.03 last 78.48) - open/close 3971 (time 1761320202.92 total 50.04 last 82.38) - open/close 4772 (time 1761320212.92 total 60.04 last 80.07) - open/close 5537 (time 1761320222.94 total 70.06 last 76.35) - open/close 6342 (time 1761320232.95 total 80.07 last 80.45) - open/close 7139 (time 1761320242.95 total 90.07 last 79.67) - open/close 7933 (time 1761320252.96 total 100.08 last 79.33) - open/close 8746 (time 1761320262.98 total 110.09 last 81.16) - open/close 9539 (time 1761320272.98 total 120.10 last 79.27) total: 10000 open/close in 125.47 seconds: 79.70 ops/second - unlinked 0 (time 1761320281 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 856 (time 1761320357.66 total 10.00 last 85.56) - open/close 1693 (time 1761320367.66 total 20.01 last 83.66) - open/close 2644 (time 1761320377.67 total 30.02 last 94.98) - open/close 3509 (time 1761320387.67 total 40.02 last 86.49) - open/close 4368 (time 1761320397.69 total 50.03 last 85.81) - open/close 5238 (time 1761320407.69 total 60.04 last 86.92) - open/close 6152 (time 1761320417.70 total 70.04 last 91.39) - open/close 6984 (time 1761320427.70 total 80.05 last 83.17) - open/close 7733 (time 1761320437.71 total 90.06 last 74.84) - open/close 8550 (time 1761320447.71 total 100.06 last 81.66) - open/close 9360 (time 1761320457.72 total 110.07 last 80.93) total: 10000 open/close in 116.78 seconds: 85.63 ops/second - unlinked 0 (time 1761320467 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock oleg114-server@tcp:/lustre /mnt/lustre On OST0, 2701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg114-server: LNET ready to unload oleg114-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 PASS 69 (1890s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 11:43:09 (1761320589) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2240 sec ========== 11:43:52 (1761320632)