== conf-sanity test 69: replace an OST with the same index ========================================================== 23:51:40 (1781236300) start mds service on oleg446-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing load_modules_local oleg446-server: Loading modules from /home/green/git/lustre-release/lustre oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg446-server: detected 4 online CPUs by sysfs oleg446-server: MODOPTS_LIBCFS= oleg446-server: Force libcfs to create 2 CPU partitions oleg446-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg446-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg446-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 514 (time 1781236378.00 total 10.01 last 51.34) - open/close 1136 (time 1781236388.03 total 20.04 last 62.04) - open/close 1788 (time 1781236398.04 total 30.05 last 65.12) - open/close 2332 (time 1781236408.05 total 40.06 last 54.33) - open/close 2919 (time 1781236418.05 total 50.06 last 58.70) - open/close 3537 (time 1781236428.06 total 60.07 last 61.76) - open/close 4191 (time 1781236438.06 total 70.08 last 65.37) - open/close 4814 (time 1781236448.07 total 80.08 last 62.28) - open/close 5397 (time 1781236458.07 total 90.08 last 58.28) - open/close 6054 (time 1781236468.08 total 100.09 last 65.62) - open/close 6657 (time 1781236478.09 total 110.10 last 60.25) - open/close 7286 (time 1781236488.10 total 120.11 last 62.83) - open/close 7951 (time 1781236498.11 total 130.12 last 66.45) - open/close 8505 (time 1781236508.11 total 140.12 last 55.39) - open/close 9011 (time 1781236518.11 total 150.12 last 50.59) - open/close 9574 (time 1781236528.12 total 160.13 last 56.25) total: 10000 open/close in 167.49 seconds: 59.71 ops/second - unlinked 0 (time 1781236539 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 624 (time 1781236664.43 total 10.01 last 62.35) - open/close 1237 (time 1781236674.43 total 20.01 last 61.26) - open/close 1838 (time 1781236684.44 total 30.02 last 60.07) - open/close 2469 (time 1781236694.45 total 40.03 last 63.01) - open/close 3115 (time 1781236704.46 total 50.04 last 64.57) - open/close 3723 (time 1781236714.47 total 60.05 last 60.75) - open/close 4344 (time 1781236724.48 total 70.06 last 62.04) - open/close 4953 (time 1781236734.48 total 80.06 last 60.88) - open/close 5578 (time 1781236744.49 total 90.07 last 62.44) - open/close 6215 (time 1781236754.52 total 100.10 last 63.51) - open/close 6853 (time 1781236764.53 total 110.12 last 63.71) - open/close 7454 (time 1781236774.54 total 120.13 last 60.04) - open/close 8059 (time 1781236784.55 total 130.14 last 60.44) - open/close 8735 (time 1781236794.59 total 140.17 last 67.37) - open/close 9356 (time 1781236804.62 total 150.20 last 61.92) - open/close 9898 (time 1781236814.62 total 160.20 last 54.18) total: 10000 open/close in 162.10 seconds: 61.69 ops/second - unlinked 0 (time 1781236821 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 591 (time 1781236937.50 total 10.00 last 59.08) - open/close 1129 (time 1781236947.52 total 20.02 last 53.72) - open/close 1734 (time 1781236957.53 total 30.03 last 60.44) - open/close 2355 (time 1781236967.54 total 40.04 last 62.00) - open/close 2919 (time 1781236977.55 total 50.05 last 56.34) - open/close 3548 (time 1781236987.56 total 60.06 last 62.87) - open/close 4221 (time 1781236997.56 total 70.06 last 67.29) - open/close 4892 (time 1781237007.58 total 80.08 last 66.98) - open/close 5488 (time 1781237017.58 total 90.08 last 59.57) - open/close 6160 (time 1781237027.59 total 100.08 last 67.19) - open/close 6771 (time 1781237037.60 total 110.10 last 61.03) - open/close 7434 (time 1781237047.60 total 120.10 last 66.26) - open/close 8091 (time 1781237057.62 total 130.12 last 65.58) - open/close 8724 (time 1781237067.64 total 140.14 last 63.19) - open/close 9309 (time 1781237077.64 total 150.14 last 58.46) - open/close 9898 (time 1781237087.65 total 160.15 last 58.85) total: 10000 open/close in 161.70 seconds: 61.84 ops/second - unlinked 0 (time 1781237092 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 594 (time 1781237210.03 total 10.01 last 59.31) - open/close 1174 (time 1781237220.03 total 20.02 last 57.97) - open/close 1726 (time 1781237230.04 total 30.02 last 55.19) - open/close 2268 (time 1781237240.04 total 40.02 last 54.19) - open/close 2770 (time 1781237250.04 total 50.02 last 50.20) - open/close 3323 (time 1781237260.04 total 60.02 last 55.30) - open/close 3887 (time 1781237270.06 total 70.04 last 56.29) - open/close 4467 (time 1781237280.06 total 80.05 last 57.97) - open/close 5095 (time 1781237290.08 total 90.07 last 62.69) - open/close 5676 (time 1781237300.09 total 100.07 last 58.06) - open/close 6269 (time 1781237310.09 total 110.08 last 59.28) - open/close 6808 (time 1781237320.10 total 120.09 last 53.84) - open/close 7355 (time 1781237330.11 total 130.10 last 54.62) - open/close 7904 (time 1781237340.13 total 140.12 last 54.80) - open/close 8480 (time 1781237350.15 total 150.13 last 57.52) - open/close 9105 (time 1781237360.15 total 160.14 last 62.48) - open/close 9672 (time 1781237370.15 total 170.14 last 56.70) total: 10000 open/close in 176.73 seconds: 56.58 ops/second - unlinked 0 (time 1781237380 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 629 (time 1781237500.95 total 10.01 last 62.83) - open/close 1179 (time 1781237510.95 total 20.02 last 54.96) - open/close 1768 (time 1781237520.96 total 30.03 last 58.86) - open/close 2395 (time 1781237530.98 total 40.04 last 62.60) - open/close 3005 (time 1781237540.98 total 50.04 last 60.98) - open/close 3594 (time 1781237550.98 total 60.05 last 58.88) - open/close 4196 (time 1781237561.00 total 70.07 last 60.07) - open/close 4775 (time 1781237571.02 total 80.08 last 57.83) - open/close 5418 (time 1781237581.03 total 90.09 last 64.25) - open/close 6056 (time 1781237591.03 total 100.10 last 63.77) - open/close 6683 (time 1781237601.04 total 110.11 last 62.63) - open/close 7316 (time 1781237611.05 total 120.11 last 63.27) - open/close 7915 (time 1781237621.05 total 130.12 last 59.88) - open/close 8450 (time 1781237631.06 total 140.13 last 53.43) - open/close 9097 (time 1781237641.08 total 150.15 last 64.59) - open/close 9664 (time 1781237651.10 total 160.17 last 56.58) total: 10000 open/close in 167.30 seconds: 59.77 ops/second - unlinked 0 (time 1781237662 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 546 (time 1781237793.02 total 10.01 last 54.54) - open/close 1142 (time 1781237803.02 total 20.02 last 59.57) - open/close 1785 (time 1781237813.03 total 30.02 last 64.28) - open/close 2327 (time 1781237823.05 total 40.04 last 54.08) - open/close 2911 (time 1781237833.06 total 50.06 last 58.31) - open/close 3527 (time 1781237843.08 total 60.07 last 61.51) - open/close 4084 (time 1781237853.08 total 70.08 last 55.66) - open/close 4591 (time 1781237863.09 total 80.09 last 50.67) - open/close 5177 (time 1781237873.10 total 90.10 last 58.54) - open/close 5707 (time 1781237883.10 total 100.10 last 52.98) - open/close 6335 (time 1781237893.11 total 110.11 last 62.75) - open/close 6926 (time 1781237903.12 total 120.12 last 59.04) - open/close 7535 (time 1781237913.12 total 130.12 last 60.90) - open/close 8194 (time 1781237923.14 total 140.13 last 65.80) - open/close 8755 (time 1781237933.14 total 150.14 last 56.07) - open/close 9364 (time 1781237943.15 total 160.15 last 60.83) - open/close 9986 (time 1781237953.16 total 170.16 last 62.14) total: 10000 open/close in 170.35 seconds: 58.70 ops/second - unlinked 0 (time 1781237957 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 589 (time 1781238083.52 total 10.01 last 58.87) - open/close 1197 (time 1781238093.54 total 20.02 last 60.73) - open/close 1764 (time 1781238103.55 total 30.03 last 56.63) - open/close 2352 (time 1781238113.56 total 40.04 last 58.71) - open/close 2944 (time 1781238123.57 total 50.05 last 59.17) - open/close 3525 (time 1781238133.57 total 60.06 last 58.05) - open/close 4133 (time 1781238143.65 total 70.13 last 60.34) - open/close 4688 (time 1781238153.66 total 80.14 last 55.45) - open/close 5302 (time 1781238163.66 total 90.15 last 61.37) - open/close 5928 (time 1781238173.67 total 100.15 last 62.59) - open/close 6448 (time 1781238183.68 total 110.16 last 51.95) - open/close 7032 (time 1781238193.68 total 120.16 last 58.40) - open/close 7615 (time 1781238203.70 total 130.18 last 58.15) - open/close 8149 (time 1781238213.70 total 140.18 last 53.40) - open/close 8621 (time 1781238223.71 total 150.19 last 47.17) - open/close 9231 (time 1781238233.72 total 160.20 last 60.93) - open/close 9792 (time 1781238243.72 total 170.20 last 56.09) total: 10000 open/close in 173.71 seconds: 57.57 ops/second - unlinked 0 (time 1781238252 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 564 (time 1781238378.03 total 10.02 last 56.31) - open/close 1147 (time 1781238388.04 total 20.03 last 58.24) - open/close 1764 (time 1781238398.05 total 30.03 last 61.65) - open/close 2386 (time 1781238408.05 total 40.04 last 62.18) - open/close 2981 (time 1781238418.05 total 50.04 last 59.48) - open/close 3558 (time 1781238428.06 total 60.04 last 57.68) - open/close 4167 (time 1781238438.06 total 70.05 last 60.87) - open/close 4760 (time 1781238448.06 total 80.05 last 59.29) - open/close 5348 (time 1781238458.07 total 90.06 last 58.77) - open/close 5942 (time 1781238468.09 total 100.08 last 59.27) - open/close 6471 (time 1781238478.10 total 110.08 last 52.87) - open/close 7001 (time 1781238488.10 total 120.09 last 52.98) - open/close 7499 (time 1781238498.12 total 130.10 last 49.73) - open/close 8014 (time 1781238508.12 total 140.11 last 51.46) - open/close 8552 (time 1781238518.13 total 150.11 last 53.78) - open/close 9008 (time 1781238528.14 total 160.12 last 45.55) - open/close 9524 (time 1781238538.14 total 170.13 last 51.57) total: 10000 open/close in 178.00 seconds: 56.18 ops/second - unlinked 0 (time 1781238550 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 609 (time 1781238671.96 total 10.02 last 60.80) - open/close 1189 (time 1781238681.96 total 20.02 last 57.98) - open/close 1765 (time 1781238691.97 total 30.03 last 57.57) - open/close 2312 (time 1781238701.97 total 40.03 last 54.67) - open/close 2896 (time 1781238711.98 total 50.04 last 58.35) - open/close 3504 (time 1781238721.99 total 60.05 last 60.74) - open/close 4101 (time 1781238732.00 total 70.06 last 59.64) - open/close 4668 (time 1781238742.00 total 80.06 last 56.69) - open/close 5224 (time 1781238752.01 total 90.07 last 55.57) - open/close 5716 (time 1781238762.03 total 100.08 last 49.11) - open/close 6251 (time 1781238772.04 total 110.10 last 53.44) - open/close 6791 (time 1781238782.04 total 120.10 last 53.98) - open/close 7380 (time 1781238792.07 total 130.13 last 58.75) - open/close 8009 (time 1781238802.09 total 140.15 last 62.72) - open/close 8558 (time 1781238812.11 total 150.17 last 54.82) - open/close 9161 (time 1781238822.11 total 160.17 last 60.27) - open/close 9750 (time 1781238832.12 total 170.18 last 58.87) total: 10000 open/close in 174.39 seconds: 57.34 ops/second - unlinked 0 (time 1781238841 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 629 (time 1781238962.20 total 10.01 last 62.86) - open/close 1240 (time 1781238972.20 total 20.01 last 61.06) - open/close 1855 (time 1781238982.22 total 30.03 last 61.41) - open/close 2439 (time 1781238992.22 total 40.03 last 58.39) - open/close 3038 (time 1781239002.22 total 50.03 last 59.88) - open/close 3661 (time 1781239012.22 total 60.04 last 62.29) - open/close 4193 (time 1781239022.23 total 70.04 last 53.17) - open/close 4698 (time 1781239032.25 total 80.06 last 50.42) - open/close 5270 (time 1781239042.26 total 90.07 last 57.14) - open/close 5942 (time 1781239052.26 total 100.07 last 67.19) - open/close 6492 (time 1781239062.27 total 110.08 last 54.93) - open/close 7057 (time 1781239072.28 total 120.09 last 56.46) - open/close 7679 (time 1781239082.30 total 130.11 last 62.09) - open/close 8279 (time 1781239092.32 total 140.13 last 59.85) - open/close 8851 (time 1781239102.33 total 150.14 last 57.13) - open/close 9442 (time 1781239112.34 total 160.15 last 59.04) total: 10000 open/close in 169.10 seconds: 59.14 ops/second - unlinked 0 (time 1781239126 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg446-server: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg446-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre On OST0, 12822 used inodes rc=0 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.146@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.146@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.146@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.204.146@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg446-server: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg446-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server unloading modules via unload_modules_local on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing unload_modules_local oleg446-server: modules unloaded. pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2