== conf-sanity test 69: replace an OST with the same index ========================================================== 19:27:33 (1763339253) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 653 (time 1763339329.80 total 10.02 last 65.17) - open/close 1291 (time 1763339339.81 total 20.03 last 63.75) - open/close 1993 (time 1763339349.83 total 30.04 last 70.08) - open/close 2706 (time 1763339359.84 total 40.06 last 71.22) - open/close 3370 (time 1763339369.84 total 50.06 last 66.37) - open/close 4042 (time 1763339379.85 total 60.07 last 67.14) - open/close 4663 (time 1763339389.87 total 70.08 last 62.01) - open/close 5390 (time 1763339399.87 total 80.09 last 72.68) - open/close 6017 (time 1763339409.88 total 90.10 last 62.60) - open/close 6646 (time 1763339419.89 total 100.11 last 62.83) - open/close 7320 (time 1763339429.90 total 110.12 last 67.35) - open/close 8013 (time 1763339439.91 total 120.12 last 69.27) - open/close 8679 (time 1763339449.91 total 130.13 last 66.56) - open/close 9372 (time 1763339459.93 total 140.15 last 69.17) total: 10000 open/close in 149.32 seconds: 66.97 ops/second - unlinked 0 (time 1763339473 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 695 (time 1763339573.30 total 10.00 last 69.50) - open/close 1400 (time 1763339583.31 total 20.01 last 70.41) - open/close 2120 (time 1763339593.32 total 30.02 last 71.93) - open/close 2799 (time 1763339603.32 total 40.03 last 67.88) - open/close 3468 (time 1763339613.34 total 50.04 last 66.80) - open/close 4119 (time 1763339623.34 total 60.05 last 65.05) - open/close 4763 (time 1763339633.35 total 70.06 last 64.34) - open/close 5407 (time 1763339643.36 total 80.06 last 64.36) - open/close 6086 (time 1763339653.37 total 90.08 last 67.81) - open/close 6789 (time 1763339663.38 total 100.09 last 70.23) - open/close 7498 (time 1763339673.39 total 110.09 last 70.84) - open/close 8168 (time 1763339683.39 total 120.10 last 66.98) - open/close 8817 (time 1763339693.40 total 130.10 last 64.88) - open/close 9479 (time 1763339703.41 total 140.11 last 66.13) total: 10000 open/close in 148.19 seconds: 67.48 ops/second - unlinked 0 (time 1763339716 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 615 (time 1763339819.91 total 10.01 last 61.44) - open/close 1213 (time 1763339829.93 total 20.03 last 59.69) - open/close 1886 (time 1763339839.93 total 30.03 last 67.28) - open/close 2623 (time 1763339849.98 total 40.09 last 73.31) - open/close 3317 (time 1763339859.99 total 50.09 last 69.39) - open/close 3965 (time 1763339869.99 total 60.10 last 64.75) - open/close 4674 (time 1763339879.99 total 70.10 last 70.90) - open/close 5331 (time 1763339890.01 total 80.11 last 65.62) - open/close 5994 (time 1763339900.02 total 90.12 last 66.22) - open/close 6652 (time 1763339910.02 total 100.12 last 65.77) - open/close 7191 (time 1763339920.04 total 110.14 last 53.79) - open/close 7753 (time 1763339930.05 total 120.15 last 56.17) - open/close 8409 (time 1763339940.06 total 130.16 last 65.56) - open/close 9030 (time 1763339950.07 total 140.17 last 62.03) - open/close 9615 (time 1763339960.07 total 150.17 last 58.48) total: 10000 open/close in 157.18 seconds: 63.62 ops/second - unlinked 0 (time 1763339971 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 784 (time 1763340073.98 total 10.00 last 78.39) - open/close 1533 (time 1763340083.98 total 20.00 last 74.88) - open/close 2243 (time 1763340093.98 total 30.01 last 70.99) - open/close 2922 (time 1763340103.99 total 40.01 last 67.86) - open/close 3649 (time 1763340113.99 total 50.01 last 72.70) - open/close 4307 (time 1763340123.99 total 60.01 last 65.79) - open/close 4977 (time 1763340134.00 total 70.02 last 66.96) - open/close 5646 (time 1763340144.00 total 80.02 last 66.88) - open/close 6283 (time 1763340154.00 total 90.03 last 63.67) - open/close 6948 (time 1763340164.01 total 100.03 last 66.47) - open/close 7627 (time 1763340174.01 total 110.03 last 67.88) - open/close 8335 (time 1763340184.02 total 120.04 last 70.75) - open/close 8970 (time 1763340194.03 total 130.06 last 63.41) - open/close 9627 (time 1763340204.05 total 140.07 last 65.59) total: 10000 open/close in 145.41 seconds: 68.77 ops/second - unlinked 0 (time 1763340213 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 645 (time 1763340312.23 total 10.00 last 64.49) - open/close 1288 (time 1763340322.23 total 20.00 last 64.28) - open/close 1980 (time 1763340332.24 total 30.02 last 69.10) - open/close 2617 (time 1763340342.25 total 40.02 last 63.67) - open/close 3285 (time 1763340352.25 total 50.03 last 66.78) - open/close 3990 (time 1763340362.26 total 60.03 last 70.45) - open/close 4722 (time 1763340372.27 total 70.04 last 73.16) - open/close 5456 (time 1763340382.27 total 80.04 last 73.39) - open/close 6190 (time 1763340392.27 total 90.04 last 73.38) - open/close 6930 (time 1763340402.28 total 100.05 last 73.92) - open/close 7685 (time 1763340412.28 total 110.06 last 75.49) - open/close 8315 (time 1763340422.28 total 120.06 last 62.99) - open/close 8953 (time 1763340432.29 total 130.07 last 63.75) - open/close 9582 (time 1763340442.31 total 140.08 last 62.80) total: 10000 open/close in 147.00 seconds: 68.03 ops/second - unlinked 0 (time 1763340453 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 736 (time 1763340555.89 total 10.01 last 73.52) - open/close 1382 (time 1763340565.90 total 20.02 last 64.56) - open/close 1975 (time 1763340575.90 total 30.02 last 59.25) - open/close 2640 (time 1763340585.91 total 40.03 last 66.45) - open/close 3349 (time 1763340595.92 total 50.04 last 70.84) - open/close 4062 (time 1763340605.93 total 60.06 last 71.19) - open/close 4813 (time 1763340615.95 total 70.07 last 75.02) - open/close 5545 (time 1763340625.96 total 80.08 last 73.13) - open/close 6276 (time 1763340635.96 total 90.08 last 73.09) - open/close 6999 (time 1763340645.96 total 100.08 last 72.30) - open/close 7696 (time 1763340655.98 total 110.10 last 69.57) - open/close 8386 (time 1763340665.99 total 120.11 last 68.93) - open/close 9093 (time 1763340675.99 total 130.11 last 70.66) - open/close 9761 (time 1763340685.99 total 140.11 last 66.79) total: 10000 open/close in 143.25 seconds: 69.81 ops/second - unlinked 0 (time 1763340692 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 669 (time 1763340792.64 total 10.02 last 66.79) - open/close 1309 (time 1763340802.64 total 20.02 last 63.98) - open/close 1973 (time 1763340812.64 total 30.02 last 66.40) - open/close 2611 (time 1763340822.65 total 40.03 last 63.71) - open/close 3276 (time 1763340832.66 total 50.04 last 66.44) - open/close 4014 (time 1763340842.67 total 60.05 last 73.76) - open/close 4694 (time 1763340852.68 total 70.06 last 67.95) - open/close 5395 (time 1763340862.68 total 80.06 last 70.09) - open/close 6073 (time 1763340872.69 total 90.07 last 67.75) - open/close 6760 (time 1763340882.69 total 100.07 last 68.65) - open/close 7400 (time 1763340892.70 total 110.08 last 63.96) - open/close 8031 (time 1763340902.72 total 120.10 last 62.98) - open/close 8738 (time 1763340912.73 total 130.11 last 70.62) - open/close 9387 (time 1763340922.74 total 140.12 last 64.87) - open/close 9942 (time 1763340932.74 total 150.12 last 55.47) total: 10000 open/close in 151.05 seconds: 66.20 ops/second - unlinked 0 (time 1763340937 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 609 (time 1763341034.14 total 10.01 last 60.83) - open/close 1265 (time 1763341044.14 total 20.02 last 65.56) - open/close 1917 (time 1763341054.15 total 30.03 last 65.15) - open/close 2551 (time 1763341064.17 total 40.04 last 63.29) - open/close 3210 (time 1763341074.18 total 50.06 last 65.79) - open/close 3898 (time 1763341084.20 total 60.08 last 68.66) - open/close 4530 (time 1763341094.20 total 70.08 last 63.20) - open/close 5216 (time 1763341104.21 total 80.09 last 68.53) - open/close 5868 (time 1763341114.22 total 90.09 last 65.16) - open/close 6515 (time 1763341124.23 total 100.11 last 64.61) - open/close 7190 (time 1763341134.24 total 110.12 last 67.44) - open/close 7859 (time 1763341144.26 total 120.13 last 66.82) - open/close 8464 (time 1763341154.27 total 130.14 last 60.41) - open/close 9190 (time 1763341164.27 total 140.15 last 72.58) - open/close 9895 (time 1763341174.28 total 150.16 last 70.43) total: 10000 open/close in 151.62 seconds: 65.95 ops/second - unlinked 0 (time 1763341178 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 691 (time 1763341278.68 total 10.01 last 69.00) - open/close 1366 (time 1763341288.69 total 20.02 last 67.46) - open/close 2030 (time 1763341298.69 total 30.03 last 66.36) - open/close 2672 (time 1763341308.71 total 40.05 last 64.07) - open/close 3322 (time 1763341318.73 total 50.07 last 64.88) - open/close 3958 (time 1763341328.74 total 60.07 last 63.55) - open/close 4640 (time 1763341338.75 total 70.09 last 68.11) - open/close 5312 (time 1763341348.76 total 80.09 last 67.16) - open/close 5972 (time 1763341358.76 total 90.10 last 65.98) - open/close 6652 (time 1763341368.77 total 100.11 last 67.95) - open/close 7310 (time 1763341378.78 total 110.11 last 65.76) - open/close 7943 (time 1763341388.79 total 120.12 last 63.24) - open/close 8593 (time 1763341398.79 total 130.13 last 64.98) - open/close 9240 (time 1763341408.80 total 140.13 last 64.66) - open/close 9879 (time 1763341418.81 total 150.14 last 63.81) total: 10000 open/close in 151.74 seconds: 65.90 ops/second - unlinked 0 (time 1763341424 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 723 (time 1763341525.66 total 10.00 last 72.27) - open/close 1494 (time 1763341535.68 total 20.02 last 76.95) - open/close 2159 (time 1763341545.69 total 30.03 last 66.47) - open/close 2866 (time 1763341555.70 total 40.04 last 70.63) - open/close 3635 (time 1763341565.71 total 50.05 last 76.84) - open/close 4339 (time 1763341575.71 total 60.05 last 70.38) - open/close 5046 (time 1763341585.72 total 70.06 last 70.62) - open/close 5725 (time 1763341595.75 total 80.09 last 67.69) - open/close 6343 (time 1763341605.76 total 90.10 last 61.77) - open/close 6969 (time 1763341615.77 total 100.11 last 62.52) - open/close 7588 (time 1763341625.78 total 110.12 last 61.85) - open/close 8222 (time 1763341635.78 total 120.12 last 63.40) - open/close 8954 (time 1763341645.78 total 130.12 last 73.16) - open/close 9645 (time 1763341655.81 total 140.15 last 68.92) total: 10000 open/close in 145.35 seconds: 68.80 ops/second - unlinked 0 (time 1763341664 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg218-server: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg218-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.118@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.118@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.118@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg218-server: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg218-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2