== conf-sanity test 69: replace an OST with the same index ========================================================== 20:54:11 (1781225651) start mds service on oleg128-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg128-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg128-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 789 (time 1781225775.72 total 10.01 last 78.84) - open/close 1577 (time 1781225785.73 total 20.02 last 78.68) - open/close 2252 (time 1781225795.74 total 30.04 last 67.42) - open/close 2938 (time 1781225805.75 total 40.04 last 68.55) - open/close 3636 (time 1781225815.77 total 50.06 last 69.67) - open/close 4297 (time 1781225825.78 total 60.07 last 66.02) - open/close 4940 (time 1781225835.79 total 70.08 last 64.27) - open/close 5571 (time 1781225845.79 total 80.08 last 63.06) - open/close 6113 (time 1781225855.80 total 90.09 last 54.16) - open/close 6623 (time 1781225865.80 total 100.10 last 50.98) - open/close 7168 (time 1781225875.82 total 110.11 last 54.40) - open/close 7826 (time 1781225885.83 total 120.12 last 65.73) - open/close 8566 (time 1781225895.83 total 130.12 last 74.00) - open/close 9321 (time 1781225905.85 total 140.14 last 75.38) total: 10000 open/close in 149.13 seconds: 67.06 ops/second - unlinked 0 (time 1781225919 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 749 (time 1781226013.30 total 10.02 last 74.76) - open/close 1539 (time 1781226023.32 total 20.04 last 78.82) - open/close 2184 (time 1781226033.34 total 30.06 last 64.40) - open/close 2876 (time 1781226043.35 total 40.07 last 69.10) - open/close 3584 (time 1781226053.36 total 50.08 last 70.73) - open/close 4244 (time 1781226063.37 total 60.09 last 65.95) - open/close 4928 (time 1781226073.38 total 70.10 last 68.33) - open/close 5709 (time 1781226083.39 total 80.11 last 78.01) - open/close 6360 (time 1781226093.40 total 90.12 last 65.05) - open/close 7020 (time 1781226103.40 total 100.12 last 65.98) - open/close 7931 (time 1781226113.41 total 110.13 last 90.99) - open/close 8512 (time 1781226123.43 total 120.15 last 58.01) - open/close 9227 (time 1781226133.44 total 130.16 last 71.44) total: 10000 open/close in 139.96 seconds: 71.45 ops/second - unlinked 0 (time 1781226147 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 942 (time 1781226241.28 total 10.00 last 94.18) - open/close 1847 (time 1781226251.28 total 20.00 last 90.48) - open/close 2619 (time 1781226261.31 total 30.03 last 77.02) - open/close 3416 (time 1781226271.33 total 40.05 last 79.54) - open/close 4323 (time 1781226281.34 total 50.06 last 90.60) - open/close 5232 (time 1781226291.34 total 60.06 last 90.85) - open/close 6097 (time 1781226301.35 total 70.07 last 86.43) - open/close 6897 (time 1781226311.35 total 80.07 last 79.99) - open/close 7583 (time 1781226321.37 total 90.09 last 68.51) - open/close 8297 (time 1781226331.37 total 100.09 last 71.37) - open/close 9059 (time 1781226341.40 total 110.12 last 75.98) - open/close 9688 (time 1781226351.41 total 120.13 last 62.86) total: 10000 open/close in 123.98 seconds: 80.66 ops/second - unlinked 0 (time 1781226360 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 707 (time 1781226452.11 total 10.05 last 70.37) - open/close 1454 (time 1781226462.11 total 20.05 last 74.69) - open/close 2274 (time 1781226472.12 total 30.06 last 81.92) - open/close 3056 (time 1781226482.14 total 40.08 last 78.05) - open/close 3822 (time 1781226492.15 total 50.09 last 76.49) - open/close 4761 (time 1781226502.16 total 60.10 last 93.81) - open/close 5427 (time 1781226512.17 total 70.11 last 66.56) - open/close 6233 (time 1781226522.18 total 80.12 last 80.47) - open/close 7225 (time 1781226532.19 total 90.13 last 99.17) - open/close 7890 (time 1781226542.20 total 100.14 last 66.39) - open/close 8555 (time 1781226552.21 total 110.15 last 66.45) - open/close 9397 (time 1781226562.22 total 120.16 last 84.13) total: 10000 open/close in 126.68 seconds: 78.94 ops/second - unlinked 0 (time 1781226572 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 765 (time 1781226663.91 total 10.00 last 76.47) - open/close 1801 (time 1781226673.92 total 20.01 last 103.50) - open/close 2478 (time 1781226683.92 total 30.02 last 67.66) - open/close 3412 (time 1781226693.93 total 40.02 last 93.37) - open/close 4143 (time 1781226703.93 total 50.03 last 73.06) - open/close 4762 (time 1781226713.94 total 60.03 last 61.87) - open/close 5453 (time 1781226723.95 total 70.05 last 69.00) - open/close 6173 (time 1781226733.97 total 80.07 last 71.83) - open/close 6915 (time 1781226743.97 total 90.07 last 74.20) - open/close 7524 (time 1781226753.99 total 100.09 last 60.81) - open/close 8215 (time 1781226763.99 total 110.09 last 69.08) - open/close 8987 (time 1781226774.01 total 120.11 last 77.06) - open/close 9777 (time 1781226784.02 total 130.11 last 78.95) total: 10000 open/close in 132.70 seconds: 75.36 ops/second - unlinked 0 (time 1781226792 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 798 (time 1781226882.93 total 10.00 last 79.78) - open/close 1723 (time 1781226892.94 total 20.01 last 92.40) - open/close 2433 (time 1781226902.96 total 30.03 last 70.91) - open/close 3302 (time 1781226912.97 total 40.04 last 86.78) - open/close 4188 (time 1781226922.97 total 50.04 last 88.59) - open/close 4999 (time 1781226932.97 total 60.05 last 81.06) - open/close 5654 (time 1781226942.98 total 70.05 last 65.45) - open/close 6304 (time 1781226953.00 total 80.07 last 64.90) - open/close 7044 (time 1781226963.00 total 90.08 last 73.95) - open/close 7858 (time 1781226973.01 total 100.08 last 81.35) - open/close 8456 (time 1781226983.01 total 110.08 last 59.79) - open/close 9280 (time 1781226993.03 total 120.10 last 82.30) total: 10000 open/close in 128.91 seconds: 77.57 ops/second - unlinked 0 (time 1781227006 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 810 (time 1781227098.13 total 10.02 last 80.86) - open/close 1590 (time 1781227108.13 total 20.02 last 78.00) - open/close 2448 (time 1781227118.13 total 30.02 last 85.78) - open/close 3192 (time 1781227128.14 total 40.02 last 74.36) - open/close 4004 (time 1781227138.14 total 50.03 last 81.14) - open/close 4725 (time 1781227148.15 total 60.04 last 72.07) - open/close 5373 (time 1781227158.16 total 70.04 last 64.74) - open/close 6084 (time 1781227168.16 total 80.05 last 71.08) - open/close 6793 (time 1781227178.17 total 90.06 last 70.84) - open/close 7460 (time 1781227188.18 total 100.07 last 66.63) - open/close 8113 (time 1781227198.18 total 110.07 last 65.26) - open/close 8783 (time 1781227208.20 total 120.09 last 66.87) - open/close 9447 (time 1781227218.21 total 130.09 last 66.39) - open/close 9973 (time 1781227228.21 total 140.10 last 52.59) total: 10000 open/close in 140.43 seconds: 71.21 ops/second - unlinked 0 (time 1781227234 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 759 (time 1781227323.57 total 10.01 last 75.80) - open/close 1535 (time 1781227333.57 total 20.01 last 77.59) - open/close 2231 (time 1781227343.57 total 30.02 last 69.58) - open/close 2954 (time 1781227353.57 total 40.02 last 72.30) - open/close 3656 (time 1781227363.58 total 50.02 last 70.16) - open/close 4307 (time 1781227373.58 total 60.03 last 65.06) - open/close 4916 (time 1781227383.59 total 70.04 last 60.83) - open/close 5638 (time 1781227393.60 total 80.05 last 72.13) - open/close 6443 (time 1781227403.61 total 90.06 last 80.42) - open/close 7146 (time 1781227413.62 total 100.07 last 70.27) - open/close 7886 (time 1781227423.62 total 110.07 last 73.96) - open/close 8638 (time 1781227433.64 total 120.08 last 75.11) - open/close 9326 (time 1781227443.64 total 130.08 last 68.80) total: 10000 open/close in 139.15 seconds: 71.86 ops/second - unlinked 0 (time 1781227457 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 718 (time 1781227556.57 total 10.01 last 71.72) - open/close 1494 (time 1781227566.57 total 20.01 last 77.58) - open/close 2279 (time 1781227576.58 total 30.02 last 78.46) - open/close 3072 (time 1781227586.58 total 40.03 last 79.23) - open/close 3959 (time 1781227596.60 total 50.04 last 88.60) - open/close 5016 (time 1781227606.60 total 60.04 last 105.66) - open/close 5869 (time 1781227616.60 total 70.04 last 85.29) - open/close 6675 (time 1781227626.61 total 80.05 last 80.53) - open/close 7297 (time 1781227636.62 total 90.07 last 62.12) - open/close 7905 (time 1781227646.64 total 100.08 last 60.71) - open/close 8614 (time 1781227656.65 total 110.09 last 70.84) - open/close 9339 (time 1781227666.65 total 120.09 last 72.49) total: 10000 open/close in 129.12 seconds: 77.45 ops/second - unlinked 0 (time 1781227682 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 672 (time 1781227783.78 total 10.01 last 67.15) - open/close 1532 (time 1781227793.80 total 20.02 last 85.90) - open/close 2191 (time 1781227803.80 total 30.02 last 65.87) - open/close 2907 (time 1781227813.81 total 40.03 last 71.54) - open/close 3568 (time 1781227823.82 total 50.04 last 66.06) - open/close 4297 (time 1781227833.82 total 60.05 last 72.83) - open/close 5103 (time 1781227843.83 total 70.06 last 80.53) - open/close 5807 (time 1781227853.83 total 80.06 last 70.40) - open/close 6556 (time 1781227863.86 total 90.08 last 74.69) - open/close 7237 (time 1781227873.87 total 100.09 last 68.04) - open/close 7907 (time 1781227883.88 total 110.10 last 66.93) - open/close 8672 (time 1781227893.88 total 120.11 last 76.49) - open/close 9380 (time 1781227903.89 total 130.11 last 70.77) total: 10000 open/close in 139.07 seconds: 71.91 ops/second - unlinked 0 (time 1781227918 ; total 1 ; last 1) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.128@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg128-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg128-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.128@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.128@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.128@tcp sys.timeout=20 start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg128-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg128-server: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg128-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: modules unloaded. pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2