-----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:52:19 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg350-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg350-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg350-server oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:54:44 (1763153684) start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg350-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 45 (195s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:58:00 (1763153880) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 629 (time 1763153949.58 total 10.01 last 62.84) - open/close 1296 (time 1763153959.58 total 20.01 last 66.69) - open/close 1976 (time 1763153969.59 total 30.02 last 67.93) - open/close 2735 (time 1763153979.59 total 40.02 last 75.90) - open/close 3461 (time 1763153989.59 total 50.02 last 72.60) - open/close 4181 (time 1763153999.60 total 60.04 last 71.90) - open/close 4926 (time 1763154009.61 total 70.04 last 74.48) - open/close 5680 (time 1763154019.62 total 80.05 last 75.28) - open/close 6370 (time 1763154029.62 total 90.06 last 68.98) - open/close 7083 (time 1763154039.63 total 100.06 last 71.28) - open/close 7808 (time 1763154049.64 total 110.07 last 72.44) - open/close 8513 (time 1763154059.64 total 120.08 last 70.43) - open/close 9208 (time 1763154069.65 total 130.08 last 69.46) - open/close 9962 (time 1763154079.65 total 140.08 last 75.39) total: 10000 open/close in 140.66 seconds: 71.10 ops/second - unlinked 0 (time 1763154083 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 723 (time 1763154174.97 total 10.01 last 72.24) - open/close 1408 (time 1763154184.97 total 20.01 last 68.45) - open/close 2119 (time 1763154194.98 total 30.02 last 71.06) - open/close 2853 (time 1763154204.98 total 40.02 last 73.37) - open/close 3524 (time 1763154214.99 total 50.03 last 67.04) - open/close 4284 (time 1763154225.00 total 60.04 last 75.92) - open/close 4987 (time 1763154235.00 total 70.04 last 70.30) - open/close 5745 (time 1763154245.01 total 80.05 last 75.74) - open/close 6469 (time 1763154255.01 total 90.05 last 72.39) - open/close 7213 (time 1763154265.01 total 100.05 last 74.38) - open/close 7879 (time 1763154275.02 total 110.06 last 66.56) - open/close 8593 (time 1763154285.02 total 120.06 last 71.40) - open/close 9401 (time 1763154295.02 total 130.06 last 80.77) total: 10000 open/close in 137.73 seconds: 72.61 ops/second - unlinked 0 (time 1763154306 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 807 (time 1763154391.60 total 10.00 last 80.68) - open/close 1605 (time 1763154401.61 total 20.01 last 79.73) - open/close 2297 (time 1763154411.63 total 30.02 last 69.11) - open/close 3004 (time 1763154421.63 total 40.03 last 70.65) - open/close 3695 (time 1763154431.65 total 50.05 last 68.97) - open/close 4433 (time 1763154441.66 total 60.05 last 73.76) - open/close 5191 (time 1763154451.66 total 70.06 last 75.77) - open/close 5939 (time 1763154461.67 total 80.06 last 74.76) - open/close 6643 (time 1763154471.67 total 90.07 last 70.38) - open/close 7258 (time 1763154481.69 total 100.08 last 61.39) - open/close 7917 (time 1763154491.69 total 110.09 last 65.86) - open/close 8651 (time 1763154501.70 total 120.10 last 73.33) - open/close 9286 (time 1763154511.71 total 130.11 last 63.42) - open/close 9905 (time 1763154521.72 total 140.12 last 61.85) total: 10000 open/close in 141.55 seconds: 70.65 ops/second - unlinked 0 (time 1763154528 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 767 (time 1763154627.52 total 10.02 last 76.54) - open/close 1464 (time 1763154637.52 total 20.02 last 69.69) - open/close 2171 (time 1763154647.52 total 30.02 last 70.69) - open/close 2902 (time 1763154657.52 total 40.02 last 73.09) - open/close 3586 (time 1763154667.53 total 50.03 last 68.36) - open/close 4204 (time 1763154677.54 total 60.04 last 61.75) - open/close 4899 (time 1763154687.54 total 70.04 last 69.47) - open/close 5635 (time 1763154697.55 total 80.05 last 73.55) - open/close 6215 (time 1763154707.55 total 90.06 last 57.96) - open/close 6854 (time 1763154717.56 total 100.07 last 63.83) - open/close 7520 (time 1763154727.57 total 110.07 last 66.55) - open/close 8197 (time 1763154737.58 total 120.09 last 67.63) - open/close 8895 (time 1763154747.59 total 130.09 last 69.75) - open/close 9577 (time 1763154757.59 total 140.09 last 68.19) total: 10000 open/close in 145.85 seconds: 68.56 ops/second - unlinked 0 (time 1763154767 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 722 (time 1763154865.26 total 10.01 last 72.10) - open/close 1442 (time 1763154875.32 total 20.07 last 71.62) - open/close 2154 (time 1763154885.33 total 30.09 last 71.06) - open/close 2856 (time 1763154895.34 total 40.09 last 70.15) - open/close 3591 (time 1763154905.35 total 50.10 last 73.45) - open/close 4306 (time 1763154915.36 total 60.11 last 71.44) - open/close 5035 (time 1763154925.36 total 70.11 last 72.86) - open/close 5766 (time 1763154935.39 total 80.14 last 72.88) - open/close 6459 (time 1763154945.39 total 90.15 last 69.27) - open/close 7174 (time 1763154955.40 total 100.15 last 71.48) - open/close 7883 (time 1763154965.40 total 110.16 last 70.86) - open/close 8634 (time 1763154975.42 total 120.17 last 75.01) - open/close 9390 (time 1763154985.42 total 130.17 last 75.59) total: 10000 open/close in 139.56 seconds: 71.65 ops/second - unlinked 0 (time 1763154999 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 793 (time 1763155091.40 total 10.02 last 79.17) - open/close 1444 (time 1763155101.40 total 20.02 last 65.08) - open/close 2144 (time 1763155111.42 total 30.04 last 69.87) - open/close 2880 (time 1763155121.42 total 40.04 last 73.57) - open/close 3611 (time 1763155131.42 total 50.04 last 73.08) - open/close 4347 (time 1763155141.43 total 60.05 last 73.54) - open/close 5024 (time 1763155151.44 total 70.06 last 67.68) - open/close 5721 (time 1763155161.45 total 80.07 last 69.63) - open/close 6452 (time 1763155171.45 total 90.07 last 73.05) - open/close 7193 (time 1763155181.47 total 100.09 last 74.00) - open/close 7953 (time 1763155191.47 total 110.09 last 75.96) - open/close 8695 (time 1763155201.48 total 120.10 last 74.11) - open/close 9389 (time 1763155211.49 total 130.11 last 69.37) total: 10000 open/close in 139.66 seconds: 71.60 ops/second - unlinked 0 (time 1763155225 ; total 1 ; last 1) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 734 (time 1763155322.91 total 10.00 last 73.37) - open/close 1405 (time 1763155332.93 total 20.03 last 66.94) - open/close 2168 (time 1763155342.94 total 30.04 last 76.21) - open/close 2885 (time 1763155352.95 total 40.05 last 71.65) - open/close 3604 (time 1763155362.95 total 50.05 last 71.89) - open/close 4282 (time 1763155372.95 total 60.05 last 67.79) - open/close 5010 (time 1763155382.95 total 70.05 last 72.79) - open/close 5727 (time 1763155392.96 total 80.06 last 71.63) - open/close 6469 (time 1763155402.97 total 90.06 last 74.17) - open/close 7216 (time 1763155412.97 total 100.06 last 74.70) - open/close 7937 (time 1763155422.98 total 110.07 last 72.03) - open/close 8592 (time 1763155432.98 total 120.08 last 65.46) - open/close 9241 (time 1763155443.01 total 130.11 last 64.73) - open/close 9930 (time 1763155453.02 total 140.11 last 68.85) total: 10000 open/close in 141.03 seconds: 70.91 ops/second - unlinked 0 (time 1763155458 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 738 (time 1763155552.75 total 10.01 last 73.76) - open/close 1448 (time 1763155562.76 total 20.01 last 70.94) - open/close 2163 (time 1763155572.77 total 30.02 last 71.45) - open/close 2904 (time 1763155582.78 total 40.03 last 74.04) - open/close 3634 (time 1763155592.78 total 50.03 last 72.99) - open/close 4365 (time 1763155602.78 total 60.03 last 73.09) - open/close 5171 (time 1763155612.78 total 70.03 last 80.59) - open/close 5904 (time 1763155622.79 total 80.04 last 73.23) - open/close 6581 (time 1763155632.80 total 90.05 last 67.65) - open/close 7280 (time 1763155642.80 total 100.05 last 69.89) - open/close 8027 (time 1763155652.81 total 110.06 last 74.63) - open/close 8759 (time 1763155662.82 total 120.07 last 73.13) - open/close 9523 (time 1763155672.82 total 130.07 last 76.37) total: 10000 open/close in 136.55 seconds: 73.23 ops/second - unlinked 0 (time 1763155683 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 723 (time 1763155781.16 total 10.00 last 72.26) - open/close 1489 (time 1763155791.18 total 20.03 last 76.44) - open/close 2278 (time 1763155801.19 total 30.03 last 78.87) - open/close 3017 (time 1763155811.21 total 40.05 last 73.76) - open/close 3755 (time 1763155821.22 total 50.06 last 73.72) - open/close 4516 (time 1763155831.24 total 60.08 last 75.93) - open/close 5283 (time 1763155841.25 total 70.09 last 76.62) - open/close 6054 (time 1763155851.27 total 80.11 last 76.94) - open/close 6809 (time 1763155861.28 total 90.12 last 75.45) - open/close 7593 (time 1763155871.29 total 100.13 last 78.34) - open/close 8370 (time 1763155881.29 total 110.13 last 77.68) - open/close 9111 (time 1763155891.29 total 120.13 last 74.05) - open/close 9892 (time 1763155901.31 total 130.15 last 78.01) total: 10000 open/close in 131.52 seconds: 76.03 ops/second - unlinked 0 (time 1763155906 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 792 (time 1763155998.16 total 10.01 last 79.13) - open/close 1534 (time 1763156008.17 total 20.01 last 74.16) - open/close 2336 (time 1763156018.17 total 30.02 last 80.15) - open/close 3080 (time 1763156028.18 total 40.03 last 74.32) - open/close 3726 (time 1763156038.20 total 50.05 last 64.50) - open/close 4414 (time 1763156048.22 total 60.06 last 68.69) - open/close 5092 (time 1763156058.22 total 70.07 last 67.77) - open/close 5799 (time 1763156068.23 total 80.08 last 70.60) - open/close 6539 (time 1763156078.25 total 90.10 last 73.89) - open/close 7279 (time 1763156088.26 total 100.10 last 73.94) - open/close 7950 (time 1763156098.26 total 110.11 last 67.07) - open/close 8697 (time 1763156108.27 total 120.12 last 74.63) - open/close 9451 (time 1763156118.28 total 130.12 last 75.35) total: 10000 open/close in 137.75 seconds: 72.60 ops/second - unlinked 0 (time 1763156129 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre On OST0, 13765 used inodes rc=0 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.150@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.150@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.150@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.150@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg350-server: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg350-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg350-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 PASS 69 (2431s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:38:32 (1763156312) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (3s) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2829 sec ========== 16:39:30 (1763156370) === conf-sanity: start cleanup 16:39:32 (1763156372) === === conf-sanity: finish cleanup 16:39:34 (1763156374) ===