== conf-sanity test 69: replace an OST with the same index ========================================================== 20:16:02 (1763342162) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 724 (time 1763342230.12 total 10.00 last 72.39) - open/close 1502 (time 1763342240.12 total 20.01 last 77.77) - open/close 2229 (time 1763342250.13 total 30.01 last 72.66) - open/close 2905 (time 1763342260.13 total 40.01 last 67.58) - open/close 3702 (time 1763342270.13 total 50.02 last 79.66) - open/close 4452 (time 1763342280.14 total 60.03 last 74.92) - open/close 5189 (time 1763342290.16 total 70.04 last 73.60) - open/close 5990 (time 1763342300.16 total 80.05 last 80.07) - open/close 6765 (time 1763342310.16 total 90.05 last 77.49) - open/close 7491 (time 1763342320.16 total 100.05 last 72.60) - open/close 8239 (time 1763342330.17 total 110.05 last 74.76) - open/close 8979 (time 1763342340.19 total 120.07 last 73.85) - open/close 9698 (time 1763342350.19 total 130.08 last 71.86) total: 10000 open/close in 134.04 seconds: 74.60 ops/second - unlinked 0 (time 1763342358 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 729 (time 1763342444.27 total 10.01 last 72.81) - open/close 1597 (time 1763342454.27 total 20.02 last 86.75) - open/close 2378 (time 1763342464.29 total 30.03 last 78.01) - open/close 3138 (time 1763342474.29 total 40.04 last 75.93) - open/close 3874 (time 1763342484.30 total 50.04 last 73.57) - open/close 4601 (time 1763342494.30 total 60.05 last 72.67) - open/close 5369 (time 1763342504.31 total 70.05 last 76.77) - open/close 6118 (time 1763342514.32 total 80.06 last 74.83) - open/close 6853 (time 1763342524.32 total 90.07 last 73.44) - open/close 7595 (time 1763342534.33 total 100.07 last 74.16) - open/close 8360 (time 1763342544.33 total 110.08 last 76.48) - open/close 9093 (time 1763342554.34 total 120.09 last 73.23) - open/close 9824 (time 1763342564.36 total 130.10 last 72.98) total: 10000 open/close in 132.54 seconds: 75.45 ops/second - unlinked 0 (time 1763342570 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 707 (time 1763342665.95 total 10.01 last 70.63) - open/close 1447 (time 1763342675.96 total 20.02 last 73.93) - open/close 2175 (time 1763342685.97 total 30.03 last 72.76) - open/close 2896 (time 1763342695.97 total 40.03 last 72.07) - open/close 3625 (time 1763342705.98 total 50.04 last 72.86) - open/close 4366 (time 1763342715.99 total 60.05 last 74.02) - open/close 5060 (time 1763342725.99 total 70.05 last 69.39) - open/close 5726 (time 1763342736.01 total 80.07 last 66.49) - open/close 6434 (time 1763342746.01 total 90.07 last 70.78) - open/close 7128 (time 1763342756.02 total 100.07 last 69.36) - open/close 7836 (time 1763342766.04 total 110.09 last 70.67) - open/close 8426 (time 1763342776.04 total 120.09 last 59.00) - open/close 9129 (time 1763342786.06 total 130.11 last 70.15) - open/close 9847 (time 1763342796.07 total 140.13 last 71.72) total: 10000 open/close in 142.50 seconds: 70.18 ops/second - unlinked 0 (time 1763342802 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 734 (time 1763342898.25 total 10.01 last 73.34) - open/close 1482 (time 1763342908.25 total 20.01 last 74.79) - open/close 2237 (time 1763342918.26 total 30.02 last 75.44) - open/close 2921 (time 1763342928.27 total 40.03 last 68.34) - open/close 3697 (time 1763342938.27 total 50.04 last 77.53) - open/close 4434 (time 1763342948.29 total 60.05 last 73.61) - open/close 5124 (time 1763342958.29 total 70.05 last 68.99) - open/close 5906 (time 1763342968.29 total 80.05 last 78.19) - open/close 6635 (time 1763342978.29 total 90.06 last 72.85) - open/close 7329 (time 1763342988.30 total 100.06 last 69.38) - open/close 8037 (time 1763342998.31 total 110.07 last 70.74) - open/close 8771 (time 1763343008.31 total 120.07 last 73.37) - open/close 9473 (time 1763343018.31 total 130.07 last 70.19) total: 10000 open/close in 138.33 seconds: 72.29 ops/second - unlinked 0 (time 1763343030 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 734 (time 1763343124.62 total 10.01 last 73.36) - open/close 1436 (time 1763343134.63 total 20.01 last 70.14) - open/close 2084 (time 1763343144.64 total 30.03 last 64.72) - open/close 2672 (time 1763343154.64 total 40.03 last 58.79) - open/close 3317 (time 1763343164.64 total 50.03 last 64.47) - open/close 4044 (time 1763343174.65 total 60.04 last 72.62) - open/close 4782 (time 1763343184.65 total 70.04 last 73.80) - open/close 5488 (time 1763343194.66 total 80.04 last 70.58) - open/close 6232 (time 1763343204.67 total 90.05 last 74.33) - open/close 6943 (time 1763343214.67 total 100.06 last 71.08) - open/close 7684 (time 1763343224.67 total 110.06 last 74.08) - open/close 8305 (time 1763343234.68 total 120.07 last 62.03) - open/close 8992 (time 1763343244.68 total 130.07 last 68.69) - open/close 9698 (time 1763343254.68 total 140.07 last 70.60) total: 10000 open/close in 144.43 seconds: 69.24 ops/second - unlinked 0 (time 1763343263 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 768 (time 1763343353.00 total 10.02 last 76.66) - open/close 1524 (time 1763343363.02 total 20.03 last 75.47) - open/close 2223 (time 1763343373.04 total 30.05 last 69.78) - open/close 3017 (time 1763343383.04 total 40.06 last 79.35) - open/close 3757 (time 1763343393.04 total 50.06 last 73.98) - open/close 4535 (time 1763343403.05 total 60.06 last 77.78) - open/close 5308 (time 1763343413.05 total 70.07 last 77.27) - open/close 6037 (time 1763343423.06 total 80.07 last 72.87) - open/close 6785 (time 1763343433.07 total 90.09 last 74.70) - open/close 7524 (time 1763343443.07 total 100.09 last 73.90) - open/close 8246 (time 1763343453.07 total 110.09 last 72.16) - open/close 8983 (time 1763343463.08 total 120.10 last 73.64) - open/close 9771 (time 1763343473.10 total 130.12 last 78.65) total: 10000 open/close in 133.41 seconds: 74.96 ops/second - unlinked 0 (time 1763343479 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 810 (time 1763343571.82 total 10.00 last 80.96) - open/close 1588 (time 1763343581.83 total 20.01 last 77.77) - open/close 2355 (time 1763343591.84 total 30.02 last 76.64) - open/close 3106 (time 1763343601.84 total 40.02 last 75.05) - open/close 3873 (time 1763343611.85 total 50.03 last 76.67) - open/close 4599 (time 1763343621.85 total 60.03 last 72.59) - open/close 5321 (time 1763343631.87 total 70.05 last 72.04) - open/close 6034 (time 1763343641.89 total 80.07 last 71.19) - open/close 6729 (time 1763343651.89 total 90.07 last 69.49) - open/close 7484 (time 1763343661.89 total 100.07 last 75.47) - open/close 8252 (time 1763343671.89 total 110.07 last 76.80) - open/close 9012 (time 1763343681.90 total 120.08 last 75.93) - open/close 9815 (time 1763343691.91 total 130.09 last 80.20) total: 10000 open/close in 132.28 seconds: 75.60 ops/second - unlinked 0 (time 1763343697 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 765 (time 1763343782.02 total 10.00 last 76.50) - open/close 1518 (time 1763343792.02 total 20.00 last 75.30) - open/close 2349 (time 1763343802.04 total 30.01 last 83.01) - open/close 3153 (time 1763343812.04 total 40.01 last 80.37) - open/close 4020 (time 1763343822.04 total 50.02 last 86.69) - open/close 4844 (time 1763343832.05 total 60.02 last 82.35) - open/close 5630 (time 1763343842.05 total 70.03 last 78.54) - open/close 6449 (time 1763343852.07 total 80.05 last 81.77) - open/close 7188 (time 1763343862.07 total 90.05 last 73.90) - open/close 7940 (time 1763343872.07 total 100.05 last 75.19) - open/close 8692 (time 1763343882.08 total 110.06 last 75.11) - open/close 9459 (time 1763343892.09 total 120.07 last 76.62) total: 10000 open/close in 127.14 seconds: 78.66 ops/second - unlinked 0 (time 1763343902 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 753 (time 1763343990.55 total 10.01 last 75.21) - open/close 1545 (time 1763344000.56 total 20.02 last 79.11) - open/close 2253 (time 1763344010.57 total 30.03 last 70.72) - open/close 3012 (time 1763344020.58 total 40.05 last 75.81) - open/close 3741 (time 1763344030.59 total 50.05 last 72.84) - open/close 4469 (time 1763344040.60 total 60.07 last 72.72) - open/close 5183 (time 1763344050.61 total 70.07 last 71.34) - open/close 5977 (time 1763344060.62 total 80.09 last 79.31) - open/close 6748 (time 1763344070.63 total 90.09 last 77.06) - open/close 7472 (time 1763344080.64 total 100.10 last 72.32) - open/close 8199 (time 1763344090.66 total 110.12 last 72.56) - open/close 8919 (time 1763344100.67 total 120.13 last 71.93) - open/close 9625 (time 1763344110.67 total 130.13 last 70.59) total: 10000 open/close in 135.76 seconds: 73.66 ops/second - unlinked 0 (time 1763344120 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 747 (time 1763344216.06 total 10.00 last 74.67) - open/close 1569 (time 1763344226.07 total 20.01 last 82.15) - open/close 2480 (time 1763344236.09 total 30.03 last 90.90) - open/close 3267 (time 1763344246.09 total 40.04 last 78.67) - open/close 4164 (time 1763344256.10 total 50.04 last 89.67) - open/close 4923 (time 1763344266.10 total 60.04 last 75.90) - open/close 5686 (time 1763344276.11 total 70.05 last 76.22) - open/close 6465 (time 1763344286.12 total 80.07 last 77.76) - open/close 7199 (time 1763344296.13 total 90.07 last 73.40) - open/close 7943 (time 1763344306.14 total 100.08 last 74.31) - open/close 8678 (time 1763344316.15 total 110.10 last 73.37) - open/close 9427 (time 1763344326.16 total 120.11 last 74.83) total: 10000 open/close in 126.99 seconds: 78.75 ops/second - unlinked 0 (time 1763344336 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.121@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.121@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg321-server: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg321-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2