== conf-sanity test 69: replace an OST with the same index ========================================================== 17:34:48 (1778535288) start mds service on oleg134-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg134-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 993 (time 1778535385.07 total 10.00 last 99.30) - open/close 1953 (time 1778535395.08 total 20.01 last 95.88) - open/close 2855 (time 1778535405.09 total 30.02 last 90.11) - open/close 3885 (time 1778535415.09 total 40.03 last 102.94) - open/close 4902 (time 1778535425.10 total 50.03 last 101.69) - open/close 6028 (time 1778535435.10 total 60.03 last 112.60) - open/close 7069 (time 1778535445.10 total 70.03 last 104.06) - open/close 8085 (time 1778535455.10 total 80.04 last 101.55) - open/close 9061 (time 1778535465.13 total 90.06 last 97.37) total: 10000 open/close in 98.89 seconds: 101.12 ops/second - unlinked 0 (time 1778535476 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1111 (time 1778535541.23 total 10.01 last 111.02) - open/close 2137 (time 1778535551.25 total 20.02 last 102.45) - open/close 3197 (time 1778535561.25 total 30.02 last 105.99) - open/close 4265 (time 1778535571.25 total 40.02 last 106.79) - open/close 5404 (time 1778535581.26 total 50.03 last 113.84) - open/close 6585 (time 1778535591.26 total 60.03 last 118.07) - open/close 7752 (time 1778535601.27 total 70.04 last 116.62) - open/close 8945 (time 1778535611.27 total 80.04 last 119.28) total: 10000 open/close in 89.67 seconds: 111.51 ops/second - unlinked 0 (time 1778535623 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1004 (time 1778535688.93 total 10.00 last 100.37) - open/close 1925 (time 1778535698.94 total 20.01 last 92.04) - open/close 2938 (time 1778535708.94 total 30.01 last 101.29) - open/close 4077 (time 1778535718.95 total 40.01 last 113.85) - open/close 5106 (time 1778535728.95 total 50.02 last 102.87) - open/close 6089 (time 1778535738.96 total 60.03 last 98.21) - open/close 7082 (time 1778535748.96 total 70.03 last 99.27) - open/close 8169 (time 1778535758.97 total 80.03 last 108.64) - open/close 9272 (time 1778535768.97 total 90.04 last 110.30) total: 10000 open/close in 96.04 seconds: 104.12 ops/second - unlinked 0 (time 1778535777 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1398 (time 1778535836.77 total 10.00 last 139.79) - open/close 2677 (time 1778535846.78 total 20.01 last 127.84) - open/close 3869 (time 1778535856.78 total 30.01 last 119.16) - open/close 4901 (time 1778535866.78 total 40.01 last 103.17) - open/close 5970 (time 1778535876.79 total 50.02 last 106.83) - open/close 7146 (time 1778535886.80 total 60.03 last 117.47) - open/close 8422 (time 1778535896.81 total 70.04 last 127.52) - open/close 9657 (time 1778535906.81 total 80.04 last 123.48) total: 10000 open/close in 83.14 seconds: 120.28 ops/second - unlinked 0 (time 1778535912 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1306 (time 1778535967.67 total 10.00 last 130.60) - open/close 2595 (time 1778535977.68 total 20.01 last 128.80) - open/close 4015 (time 1778535987.69 total 30.02 last 141.82) - open/close 5613 (time 1778535997.69 total 40.02 last 159.74) - open/close 6950 (time 1778536007.70 total 50.03 last 133.59) - open/close 8203 (time 1778536017.71 total 60.04 last 125.23) - open/close 9594 (time 1778536027.71 total 70.04 last 139.05) total: 10000 open/close in 73.06 seconds: 136.87 ops/second - unlinked 0 (time 1778536032 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1361 (time 1778536086.84 total 10.01 last 136.03) - open/close 2636 (time 1778536096.84 total 20.01 last 127.44) - open/close 3783 (time 1778536106.85 total 30.01 last 114.66) - open/close 5153 (time 1778536116.85 total 40.02 last 136.91) - open/close 6512 (time 1778536126.86 total 50.03 last 135.79) - open/close 7892 (time 1778536136.88 total 60.04 last 137.80) - open/close 9310 (time 1778536146.88 total 70.04 last 141.77) total: 10000 open/close in 75.26 seconds: 132.87 ops/second - unlinked 0 (time 1778536154 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1373 (time 1778536204.26 total 10.02 last 137.06) - open/close 2636 (time 1778536214.26 total 20.02 last 126.24) - open/close 4071 (time 1778536224.27 total 30.03 last 143.40) - open/close 5484 (time 1778536234.28 total 40.03 last 141.23) - open/close 6930 (time 1778536244.28 total 50.04 last 144.53) - open/close 8272 (time 1778536254.28 total 60.04 last 134.18) - open/close 9546 (time 1778536264.30 total 70.05 last 127.23) total: 10000 open/close in 73.49 seconds: 136.08 ops/second - unlinked 0 (time 1778536269 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1316 (time 1778536321.04 total 10.01 last 131.49) - open/close 2718 (time 1778536331.04 total 20.01 last 140.17) - open/close 4149 (time 1778536341.05 total 30.02 last 142.97) - open/close 5394 (time 1778536351.05 total 40.02 last 124.48) - open/close 6144 (time 1778536361.12 total 50.09 last 74.50) - open/close 6965 (time 1778536371.13 total 60.10 last 82.00) - open/close 7696 (time 1778536381.14 total 70.11 last 73.04) - open/close 8584 (time 1778536391.14 total 80.11 last 88.77) - open/close 9388 (time 1778536401.15 total 90.12 last 80.36) total: 10000 open/close in 97.22 seconds: 102.86 ops/second - unlinked 0 (time 1778536411 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1089 (time 1778536475.58 total 10.01 last 108.82) - open/close 2151 (time 1778536485.58 total 20.01 last 106.17) - open/close 3161 (time 1778536495.58 total 30.01 last 100.99) - open/close 4227 (time 1778536505.59 total 40.02 last 106.54) - open/close 5082 (time 1778536515.59 total 50.03 last 85.43) - open/close 6185 (time 1778536525.61 total 60.04 last 110.14) - open/close 7258 (time 1778536535.61 total 70.04 last 107.26) - open/close 8083 (time 1778536545.62 total 80.05 last 82.46) - open/close 9045 (time 1778536555.63 total 90.06 last 96.08) total: 10000 open/close in 99.72 seconds: 100.28 ops/second - unlinked 0 (time 1778536568 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 885 (time 1778536644.32 total 10.01 last 88.42) - open/close 1854 (time 1778536654.33 total 20.02 last 96.84) - open/close 2812 (time 1778536664.33 total 30.02 last 95.80) - open/close 3941 (time 1778536674.33 total 40.02 last 112.81) - open/close 4969 (time 1778536684.34 total 50.03 last 102.78) - open/close 6009 (time 1778536694.34 total 60.03 last 103.96) - open/close 7044 (time 1778536704.36 total 70.05 last 103.33) - open/close 8036 (time 1778536714.36 total 80.05 last 99.20) - open/close 9041 (time 1778536724.37 total 90.06 last 100.40) - open/close 9985 (time 1778536734.40 total 100.09 last 94.07) total: 10000 open/close in 100.28 seconds: 99.72 ops/second - unlinked 0 (time 1778536738 ; total 1 ; last 1) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.134@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.134@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg134-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre On OST0, 10519 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg134-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg134-server: LNET ready to unload oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2