== conf-sanity test 69: replace an OST with the same index ========================================================== 22:27:58 (1787624878) start mds service on oleg608-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing load_modules_local oleg608-server: Loading modules from /home/green/git/lustre-release/lustre oleg608-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg608-server: detected 4 online CPUs by sysfs oleg608-server: MODOPTS_LIBCFS= oleg608-server: Force libcfs to create 2 CPU partitions oleg608-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg608-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg608-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg608-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg608-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-OST0000 oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg608-client.virtnet: -o user_xattr,flock 192.168.206.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 848 (time 1787624982.77 total 10.01 last 84.70) - open/close 1658 (time 1787624992.77 total 20.01 last 81.00) - open/close 2467 (time 1787625002.78 total 30.01 last 80.89) - open/close 3209 (time 1787625012.78 total 40.02 last 74.15) - open/close 3806 (time 1787625022.81 total 50.04 last 59.56) - open/close 4470 (time 1787625032.81 total 60.05 last 66.36) - open/close 5246 (time 1787625042.82 total 70.06 last 77.54) - open/close 5961 (time 1787625052.84 total 80.08 last 71.37) - open/close 6640 (time 1787625062.85 total 90.09 last 67.82) - open/close 7279 (time 1787625072.85 total 100.09 last 63.89) - open/close 7989 (time 1787625082.87 total 110.10 last 70.91) - open/close 8696 (time 1787625092.87 total 120.11 last 70.65) - open/close 9459 (time 1787625102.87 total 130.11 last 76.28) total: 10000 open/close in 137.48 seconds: 72.74 ops/second - unlinked 0 (time 1787625114 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 790 (time 1787625199.21 total 10.00 last 78.99) - open/close 1599 (time 1787625209.21 total 20.00 last 80.89) - open/close 2466 (time 1787625219.21 total 30.01 last 86.64) - open/close 3295 (time 1787625229.22 total 40.02 last 82.84) - open/close 4075 (time 1787625239.23 total 50.02 last 77.97) - open/close 4704 (time 1787625249.24 total 60.04 last 62.80) - open/close 5209 (time 1787625259.25 total 70.04 last 50.47) - open/close 5784 (time 1787625269.26 total 80.06 last 57.41) - open/close 6541 (time 1787625279.27 total 90.07 last 75.64) - open/close 7374 (time 1787625289.29 total 100.08 last 83.16) - open/close 8177 (time 1787625299.30 total 110.10 last 80.16) - open/close 9022 (time 1787625309.31 total 120.10 last 84.48) - open/close 9836 (time 1787625319.31 total 130.11 last 81.36) total: 10000 open/close in 131.87 seconds: 75.83 ops/second - unlinked 0 (time 1787625325 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 776 (time 1787625407.74 total 10.01 last 77.56) - open/close 1434 (time 1787625417.76 total 20.02 last 65.70) - open/close 2098 (time 1787625427.76 total 30.02 last 66.40) - open/close 2777 (time 1787625437.77 total 40.04 last 67.80) - open/close 3445 (time 1787625447.78 total 50.04 last 66.79) - open/close 4099 (time 1787625457.78 total 60.04 last 65.36) - open/close 4942 (time 1787625467.79 total 70.05 last 84.23) - open/close 5757 (time 1787625477.79 total 80.05 last 81.48) - open/close 6613 (time 1787625487.80 total 90.07 last 85.48) - open/close 7429 (time 1787625497.81 total 100.07 last 81.59) - open/close 8250 (time 1787625507.82 total 110.09 last 81.96) - open/close 9067 (time 1787625517.83 total 120.09 last 81.69) - open/close 9891 (time 1787625527.84 total 130.10 last 82.30) total: 10000 open/close in 131.40 seconds: 76.10 ops/second - unlinked 0 (time 1787625532 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 818 (time 1787625618.00 total 10.01 last 81.70) - open/close 1558 (time 1787625628.01 total 20.02 last 73.90) - open/close 2450 (time 1787625638.02 total 30.03 last 89.15) - open/close 3279 (time 1787625648.03 total 40.04 last 82.82) - open/close 3952 (time 1787625658.04 total 50.05 last 67.26) - open/close 4512 (time 1787625668.04 total 60.05 last 56.00) - open/close 5277 (time 1787625678.04 total 70.05 last 76.43) - open/close 6026 (time 1787625688.05 total 80.06 last 74.88) - open/close 6859 (time 1787625698.06 total 90.07 last 83.22) - open/close 7696 (time 1787625708.07 total 100.08 last 83.57) - open/close 8494 (time 1787625718.08 total 110.10 last 79.70) - open/close 9289 (time 1787625728.09 total 120.10 last 79.45) total: 10000 open/close in 129.17 seconds: 77.42 ops/second - unlinked 0 (time 1787625741 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 745 (time 1787625827.88 total 10.01 last 74.46) - open/close 1584 (time 1787625837.88 total 20.01 last 83.90) - open/close 2394 (time 1787625847.89 total 30.02 last 80.90) - open/close 3220 (time 1787625857.90 total 40.03 last 82.51) - open/close 3965 (time 1787625867.91 total 50.04 last 74.41) - open/close 4804 (time 1787625877.92 total 60.04 last 83.87) - open/close 5594 (time 1787625887.92 total 70.05 last 78.99) - open/close 6458 (time 1787625897.92 total 80.05 last 86.40) - open/close 7229 (time 1787625907.93 total 90.05 last 77.06) - open/close 8046 (time 1787625917.94 total 100.06 last 81.62) - open/close 8710 (time 1787625927.94 total 110.07 last 66.35) - open/close 9502 (time 1787625937.95 total 120.08 last 79.14) total: 10000 open/close in 125.98 seconds: 79.38 ops/second - unlinked 0 (time 1787625947 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 833 (time 1787626038.32 total 10.03 last 83.09) - open/close 1686 (time 1787626048.32 total 20.03 last 85.27) - open/close 2550 (time 1787626058.32 total 30.03 last 86.37) - open/close 3332 (time 1787626068.33 total 40.04 last 78.13) - open/close 4092 (time 1787626078.34 total 50.04 last 75.97) - open/close 4718 (time 1787626088.34 total 60.04 last 62.60) - open/close 5375 (time 1787626098.34 total 70.05 last 65.65) - open/close 6153 (time 1787626108.35 total 80.06 last 77.72) - open/close 6833 (time 1787626118.37 total 90.07 last 67.91) - open/close 7621 (time 1787626128.37 total 100.08 last 78.73) - open/close 8403 (time 1787626138.38 total 110.09 last 78.14) - open/close 9233 (time 1787626148.39 total 120.10 last 82.91) total: 10000 open/close in 129.24 seconds: 77.37 ops/second - unlinked 0 (time 1787626161 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 861 (time 1787626245.12 total 10.00 last 86.07) - open/close 1677 (time 1787626255.13 total 20.01 last 81.54) - open/close 2528 (time 1787626265.13 total 30.01 last 85.07) - open/close 3318 (time 1787626275.14 total 40.02 last 78.96) - open/close 4034 (time 1787626285.14 total 50.02 last 71.60) - open/close 4799 (time 1787626295.15 total 60.03 last 76.42) - open/close 5529 (time 1787626305.17 total 70.05 last 72.85) - open/close 6327 (time 1787626315.19 total 80.07 last 79.64) - open/close 7179 (time 1787626325.20 total 90.08 last 85.09) - open/close 7957 (time 1787626335.21 total 100.09 last 77.73) - open/close 8654 (time 1787626345.22 total 110.10 last 69.65) - open/close 9460 (time 1787626355.23 total 120.11 last 80.48) total: 10000 open/close in 126.39 seconds: 79.12 ops/second - unlinked 0 (time 1787626365 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 805 (time 1787626450.68 total 10.02 last 80.34) - open/close 1601 (time 1787626460.68 total 20.03 last 79.52) - open/close 2447 (time 1787626470.69 total 30.04 last 84.52) - open/close 3208 (time 1787626480.70 total 40.05 last 76.03) - open/close 4023 (time 1787626490.73 total 50.07 last 81.31) - open/close 4842 (time 1787626500.73 total 60.08 last 81.86) - open/close 5681 (time 1787626510.74 total 70.08 last 83.82) - open/close 6472 (time 1787626520.74 total 80.09 last 79.07) - open/close 7288 (time 1787626530.76 total 90.11 last 81.45) - open/close 8162 (time 1787626540.77 total 100.11 last 87.37) - open/close 8968 (time 1787626550.77 total 110.11 last 80.59) - open/close 9845 (time 1787626560.77 total 120.11 last 87.68) total: 10000 open/close in 121.89 seconds: 82.04 ops/second - unlinked 0 (time 1787626566 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 707 (time 1787626650.74 total 10.00 last 70.68) - open/close 1495 (time 1787626660.74 total 20.01 last 78.76) - open/close 2315 (time 1787626670.75 total 30.01 last 81.97) - open/close 3124 (time 1787626680.76 total 40.03 last 80.76) - open/close 3955 (time 1787626690.78 total 50.04 last 82.98) - open/close 4801 (time 1787626700.78 total 60.05 last 84.57) - open/close 5625 (time 1787626710.80 total 70.07 last 82.25) - open/close 6481 (time 1787626720.80 total 80.07 last 85.59) - open/close 7360 (time 1787626730.81 total 90.07 last 87.88) - open/close 8208 (time 1787626740.81 total 100.08 last 84.74) - open/close 9059 (time 1787626750.82 total 110.08 last 85.07) - open/close 9930 (time 1787626760.83 total 120.09 last 87.02) total: 10000 open/close in 120.91 seconds: 82.71 ops/second - unlinked 0 (time 1787626764 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 810 (time 1787626854.04 total 10.01 last 80.95) - open/close 1604 (time 1787626864.05 total 20.01 last 79.34) - open/close 2405 (time 1787626874.05 total 30.02 last 80.08) - open/close 3213 (time 1787626884.05 total 40.02 last 80.79) - open/close 4022 (time 1787626894.06 total 50.03 last 80.78) - open/close 4859 (time 1787626904.07 total 60.04 last 83.60) - open/close 5807 (time 1787626914.08 total 70.05 last 94.77) - open/close 6740 (time 1787626924.08 total 80.05 last 93.29) - open/close 7520 (time 1787626934.09 total 90.06 last 77.88) - open/close 8384 (time 1787626944.10 total 100.07 last 86.34) - open/close 9172 (time 1787626954.11 total 110.07 last 78.76) total: 10000 open/close in 119.48 seconds: 83.69 ops/second - unlinked 0 (time 1787626967 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg608-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg608-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-OST0000 oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg608-server: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg608-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg608-server: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg608-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg608-client.virtnet: -o user_xattr,flock 192.168.206.108@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg608-client: oleg608-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg608-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.108@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.108@tcp sys.timeout=20 start ost1 service on oleg608-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-OST0000 oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg608-server: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg608-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg608-server: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg608-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg608-client.virtnet: -o user_xattr,flock 192.168.206.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg608-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg608-server unloading modules via unload_modules_local on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing unload_modules_local oleg608-server: modules unloaded. pdsh@oleg608-client: oleg608-client: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-server: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-client: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-server: ssh exited with exit code 2