== conf-sanity test 69: replace an OST with the same index ========================================================== 10:06:34 (1787580394) start mds service on oleg356-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg356-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 753 (time 1787580508.25 total 10.01 last 75.19) - open/close 1499 (time 1787580518.25 total 20.02 last 74.54) - open/close 2191 (time 1787580528.26 total 30.03 last 69.13) - open/close 2989 (time 1787580538.29 total 40.06 last 79.60) - open/close 3802 (time 1787580548.29 total 50.06 last 81.28) - open/close 4586 (time 1787580558.30 total 60.07 last 78.30) - open/close 5397 (time 1787580568.31 total 70.08 last 81.06) - open/close 6224 (time 1787580578.31 total 80.08 last 82.67) - open/close 7016 (time 1787580588.32 total 90.09 last 79.11) - open/close 7768 (time 1787580598.34 total 100.11 last 75.07) - open/close 8589 (time 1787580608.35 total 110.12 last 82.02) - open/close 9381 (time 1787580618.35 total 120.12 last 79.17) total: 10000 open/close in 127.95 seconds: 78.16 ops/second - unlinked 0 (time 1787580632 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 827 (time 1787580721.92 total 10.01 last 82.66) - open/close 1519 (time 1787580731.93 total 20.01 last 69.17) - open/close 2277 (time 1787580741.94 total 30.02 last 75.73) - open/close 2969 (time 1787580751.95 total 40.03 last 69.15) - open/close 3720 (time 1787580761.95 total 50.03 last 75.06) - open/close 4441 (time 1787580771.97 total 60.05 last 71.95) - open/close 5214 (time 1787580781.97 total 70.05 last 77.30) - open/close 5995 (time 1787580791.98 total 80.06 last 78.05) - open/close 6748 (time 1787580801.99 total 90.07 last 75.21) - open/close 7475 (time 1787580811.99 total 100.07 last 72.68) - open/close 8095 (time 1787580822.00 total 110.08 last 61.95) - open/close 8846 (time 1787580832.01 total 120.09 last 75.06) - open/close 9501 (time 1787580842.01 total 130.09 last 65.49) total: 10000 open/close in 136.83 seconds: 73.08 ops/second - unlinked 0 (time 1787580853 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 697 (time 1787580944.80 total 10.02 last 69.56) - open/close 1354 (time 1787580954.82 total 20.04 last 65.60) - open/close 2118 (time 1787580964.83 total 30.05 last 76.29) - open/close 2903 (time 1787580974.84 total 40.06 last 78.41) - open/close 3599 (time 1787580984.86 total 50.08 last 69.45) - open/close 4222 (time 1787580994.87 total 60.09 last 62.25) - open/close 4911 (time 1787581004.89 total 70.11 last 68.78) - open/close 5679 (time 1787581014.90 total 80.12 last 76.72) - open/close 6380 (time 1787581024.92 total 90.14 last 69.99) - open/close 6945 (time 1787581034.92 total 100.14 last 56.48) - open/close 7535 (time 1787581044.93 total 110.15 last 58.94) - open/close 8054 (time 1787581054.93 total 120.15 last 51.87) - open/close 8666 (time 1787581064.94 total 130.16 last 61.18) - open/close 9408 (time 1787581074.94 total 140.16 last 74.19) total: 10000 open/close in 148.62 seconds: 67.29 ops/second - unlinked 0 (time 1787581087 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 610 (time 1787581182.01 total 10.00 last 61.00) - open/close 1355 (time 1787581192.04 total 20.03 last 74.31) - open/close 2100 (time 1787581202.04 total 30.03 last 74.45) - open/close 2942 (time 1787581212.06 total 40.05 last 84.08) - open/close 3705 (time 1787581222.06 total 50.05 last 76.26) - open/close 4438 (time 1787581232.07 total 60.06 last 73.22) - open/close 5200 (time 1787581242.08 total 70.07 last 76.11) - open/close 6061 (time 1787581252.09 total 80.07 last 86.08) - open/close 6809 (time 1787581262.09 total 90.08 last 74.78) - open/close 7570 (time 1787581272.10 total 100.09 last 76.02) - open/close 8358 (time 1787581282.11 total 110.10 last 78.70) - open/close 9152 (time 1787581292.12 total 120.11 last 79.32) - open/close 9926 (time 1787581302.13 total 130.11 last 77.38) total: 10000 open/close in 131.23 seconds: 76.20 ops/second - unlinked 0 (time 1787581306 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 752 (time 1787581403.98 total 10.00 last 75.20) - open/close 1499 (time 1787581413.98 total 20.01 last 74.66) - open/close 2205 (time 1787581423.99 total 30.01 last 70.57) - open/close 2941 (time 1787581434.00 total 40.02 last 73.49) - open/close 3776 (time 1787581444.01 total 50.03 last 83.45) - open/close 4587 (time 1787581454.01 total 60.04 last 81.06) - open/close 5193 (time 1787581464.02 total 70.04 last 60.59) - open/close 5785 (time 1787581474.04 total 80.06 last 59.06) - open/close 6279 (time 1787581484.05 total 90.07 last 49.35) - open/close 6911 (time 1787581494.06 total 100.08 last 63.15) - open/close 7643 (time 1787581504.06 total 110.08 last 73.18) - open/close 8366 (time 1787581514.07 total 120.09 last 72.26) - open/close 9119 (time 1787581524.07 total 130.10 last 75.23) - open/close 9878 (time 1787581534.08 total 140.10 last 75.89) total: 10000 open/close in 141.60 seconds: 70.62 ops/second - unlinked 0 (time 1787581540 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 699 (time 1787581634.36 total 10.00 last 69.89) - open/close 1472 (time 1787581644.37 total 20.01 last 77.24) - open/close 2089 (time 1787581654.38 total 30.02 last 61.65) - open/close 2703 (time 1787581664.39 total 40.03 last 61.34) - open/close 3355 (time 1787581674.40 total 50.04 last 65.13) - open/close 4182 (time 1787581684.40 total 60.04 last 82.70) - open/close 4924 (time 1787581694.43 total 70.07 last 73.99) - open/close 5723 (time 1787581704.43 total 80.07 last 79.90) - open/close 6489 (time 1787581714.43 total 90.07 last 76.55) - open/close 7212 (time 1787581724.44 total 100.07 last 72.29) - open/close 8000 (time 1787581734.44 total 110.07 last 78.79) - open/close 8799 (time 1787581744.44 total 120.07 last 79.89) - open/close 9578 (time 1787581754.44 total 130.08 last 77.88) total: 10000 open/close in 135.96 seconds: 73.55 ops/second - unlinked 0 (time 1787581766 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 695 (time 1787581866.84 total 10.01 last 69.46) - open/close 1448 (time 1787581876.85 total 20.01 last 75.24) - open/close 2173 (time 1787581886.86 total 30.02 last 72.44) - open/close 2901 (time 1787581896.86 total 40.02 last 72.77) - open/close 3593 (time 1787581906.87 total 50.04 last 69.12) - open/close 4306 (time 1787581916.88 total 60.04 last 71.28) - open/close 5081 (time 1787581926.88 total 70.05 last 77.45) - open/close 5783 (time 1787581936.89 total 80.05 last 70.18) - open/close 6535 (time 1787581946.89 total 90.05 last 75.20) - open/close 7291 (time 1787581956.90 total 100.07 last 75.48) - open/close 8030 (time 1787581966.90 total 110.07 last 73.90) - open/close 8759 (time 1787581976.91 total 120.07 last 72.88) - open/close 9501 (time 1787581986.91 total 130.07 last 74.18) total: 10000 open/close in 137.84 seconds: 72.55 ops/second - unlinked 0 (time 1787581999 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 727 (time 1787582093.75 total 10.01 last 72.62) - open/close 1400 (time 1787582103.75 total 20.02 last 67.27) - open/close 2158 (time 1787582113.76 total 30.02 last 75.76) - open/close 2845 (time 1787582123.76 total 40.02 last 68.70) - open/close 3556 (time 1787582133.76 total 50.02 last 71.09) - open/close 4305 (time 1787582143.76 total 60.02 last 74.90) - open/close 4978 (time 1787582153.76 total 70.03 last 67.26) - open/close 5692 (time 1787582163.77 total 80.03 last 71.38) - open/close 6443 (time 1787582173.79 total 90.05 last 74.94) - open/close 7170 (time 1787582183.79 total 100.06 last 72.66) - open/close 7892 (time 1787582193.80 total 110.06 last 72.19) - open/close 8583 (time 1787582203.80 total 120.06 last 69.07) - open/close 9292 (time 1787582213.82 total 130.08 last 70.75) total: 10000 open/close in 139.89 seconds: 71.48 ops/second - unlinked 0 (time 1787582228 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 620 (time 1787582327.28 total 10.00 last 61.98) - open/close 1272 (time 1787582337.30 total 20.02 last 65.09) - open/close 1993 (time 1787582347.30 total 30.02 last 72.08) - open/close 2684 (time 1787582357.31 total 40.03 last 69.02) - open/close 3362 (time 1787582367.31 total 50.03 last 67.79) - open/close 4080 (time 1787582377.32 total 60.04 last 71.74) - open/close 4674 (time 1787582387.33 total 70.05 last 59.35) - open/close 5386 (time 1787582397.34 total 80.06 last 71.14) - open/close 6023 (time 1787582407.35 total 90.07 last 63.64) - open/close 6668 (time 1787582417.35 total 100.07 last 64.47) - open/close 7434 (time 1787582427.37 total 110.10 last 76.43) - open/close 8068 (time 1787582437.38 total 120.10 last 63.36) - open/close 8833 (time 1787582447.39 total 130.11 last 76.45) - open/close 9545 (time 1787582457.39 total 140.11 last 71.17) total: 10000 open/close in 146.35 seconds: 68.33 ops/second - unlinked 0 (time 1787582467 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 715 (time 1787582567.06 total 10.00 last 71.50) - open/close 1494 (time 1787582577.07 total 20.01 last 77.84) - open/close 2221 (time 1787582587.08 total 30.01 last 72.67) - open/close 2799 (time 1787582597.08 total 40.01 last 57.79) - open/close 3427 (time 1787582607.09 total 50.02 last 62.74) - open/close 4051 (time 1787582617.10 total 60.03 last 62.33) - open/close 4709 (time 1787582627.11 total 70.05 last 65.70) - open/close 5384 (time 1787582637.11 total 80.05 last 67.49) - open/close 6059 (time 1787582647.12 total 90.06 last 67.44) - open/close 6749 (time 1787582657.13 total 100.07 last 68.93) - open/close 7548 (time 1787582667.14 total 110.08 last 79.83) - open/close 8292 (time 1787582677.16 total 120.09 last 74.28) - open/close 8926 (time 1787582687.17 total 130.11 last 63.31) - open/close 9465 (time 1787582697.21 total 140.15 last 53.69) total: 10000 open/close in 148.66 seconds: 67.27 ops/second - unlinked 0 (time 1787582710 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: modules unloaded. pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2