== conf-sanity test 69: replace an OST with the same index ========================================================== 15:29:23 (1778614163) start mds service on oleg208-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg208-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg208-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 681 (time 1778614288.97 total 10.01 last 68.02) - open/close 1474 (time 1778614298.97 total 20.02 last 79.27) - open/close 2112 (time 1778614308.98 total 30.03 last 63.74) - open/close 2824 (time 1778614319.00 total 40.04 last 71.08) - open/close 3489 (time 1778614329.01 total 50.05 last 66.43) - open/close 4181 (time 1778614339.01 total 60.06 last 69.18) - open/close 4949 (time 1778614349.02 total 70.06 last 76.73) - open/close 5584 (time 1778614359.02 total 80.07 last 63.49) - open/close 6350 (time 1778614369.02 total 90.07 last 76.59) - open/close 7152 (time 1778614379.03 total 100.08 last 80.11) - open/close 7873 (time 1778614389.04 total 110.08 last 72.07) - open/close 8700 (time 1778614399.04 total 120.08 last 82.69) - open/close 9470 (time 1778614409.05 total 130.10 last 76.89) total: 10000 open/close in 137.69 seconds: 72.63 ops/second - unlinked 0 (time 1778614421 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 734 (time 1778614515.71 total 10.00 last 73.40) - open/close 1439 (time 1778614525.72 total 20.01 last 70.44) - open/close 2120 (time 1778614535.73 total 30.01 last 68.07) - open/close 2708 (time 1778614545.75 total 40.03 last 58.69) - open/close 3392 (time 1778614555.75 total 50.03 last 68.39) - open/close 4078 (time 1778614565.76 total 60.05 last 68.52) - open/close 4733 (time 1778614575.77 total 70.05 last 65.45) - open/close 5504 (time 1778614585.77 total 80.06 last 77.05) - open/close 6306 (time 1778614595.78 total 90.06 last 80.17) - open/close 7012 (time 1778614605.78 total 100.06 last 70.59) - open/close 7601 (time 1778614615.79 total 110.08 last 58.82) - open/close 8219 (time 1778614625.80 total 120.08 last 61.76) - open/close 8892 (time 1778614635.81 total 130.10 last 67.21) - open/close 9624 (time 1778614645.82 total 140.10 last 73.15) total: 10000 open/close in 144.61 seconds: 69.15 ops/second - unlinked 0 (time 1778614655 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 683 (time 1778614754.84 total 10.00 last 68.29) - open/close 1395 (time 1778614764.85 total 20.02 last 71.10) - open/close 2119 (time 1778614774.86 total 30.02 last 72.34) - open/close 2864 (time 1778614784.87 total 40.03 last 74.47) - open/close 3584 (time 1778614794.87 total 50.03 last 71.98) - open/close 4290 (time 1778614804.88 total 60.04 last 70.56) - open/close 4948 (time 1778614814.90 total 70.07 last 65.61) - open/close 5619 (time 1778614824.91 total 80.07 last 67.07) - open/close 6293 (time 1778614834.92 total 90.08 last 67.36) - open/close 6904 (time 1778614844.92 total 100.09 last 61.05) - open/close 7486 (time 1778614854.95 total 110.11 last 58.06) - open/close 8160 (time 1778614864.95 total 120.11 last 67.40) - open/close 8823 (time 1778614874.96 total 130.12 last 66.23) - open/close 9433 (time 1778614884.96 total 140.12 last 60.99) total: 10000 open/close in 148.08 seconds: 67.53 ops/second - unlinked 0 (time 1778614898 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 553 (time 1778615004.47 total 10.01 last 55.25) - open/close 1221 (time 1778615014.47 total 20.02 last 66.75) - open/close 1933 (time 1778615024.48 total 30.02 last 71.15) - open/close 2560 (time 1778615034.48 total 40.02 last 62.69) - open/close 3108 (time 1778615044.48 total 50.03 last 54.79) - open/close 3707 (time 1778615054.49 total 60.03 last 59.86) - open/close 4336 (time 1778615064.51 total 70.05 last 62.77) - open/close 4897 (time 1778615074.51 total 80.06 last 56.09) - open/close 5504 (time 1778615084.52 total 90.06 last 60.67) - open/close 6068 (time 1778615094.52 total 100.07 last 56.37) - open/close 6746 (time 1778615104.52 total 110.07 last 67.79) - open/close 7217 (time 1778615114.54 total 120.08 last 47.04) - open/close 7940 (time 1778615124.54 total 130.09 last 72.25) - open/close 8546 (time 1778615134.55 total 140.09 last 60.59) - open/close 9167 (time 1778615144.55 total 150.10 last 62.05) - open/close 9777 (time 1778615154.55 total 160.10 last 61.00) total: 10000 open/close in 163.07 seconds: 61.32 ops/second - unlinked 0 (time 1778615162 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 659 (time 1778615260.56 total 10.01 last 65.85) - open/close 1266 (time 1778615270.57 total 20.02 last 60.63) - open/close 2067 (time 1778615280.58 total 30.02 last 80.05) - open/close 2697 (time 1778615290.59 total 40.03 last 62.94) - open/close 3259 (time 1778615300.60 total 50.04 last 56.15) - open/close 3995 (time 1778615310.60 total 60.05 last 73.56) - open/close 4736 (time 1778615320.61 total 70.06 last 74.04) - open/close 5415 (time 1778615330.62 total 80.07 last 67.85) - open/close 6064 (time 1778615340.63 total 90.07 last 64.84) - open/close 6693 (time 1778615350.64 total 100.08 last 62.83) - open/close 7314 (time 1778615360.65 total 110.10 last 62.03) - open/close 7939 (time 1778615370.68 total 120.13 last 62.31) - open/close 8564 (time 1778615380.69 total 130.14 last 62.44) - open/close 9146 (time 1778615390.71 total 140.15 last 58.10) - open/close 9675 (time 1778615400.73 total 150.18 last 52.79) total: 10000 open/close in 155.72 seconds: 64.22 ops/second - unlinked 0 (time 1778615411 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 649 (time 1778615506.74 total 10.02 last 64.78) - open/close 1235 (time 1778615516.75 total 20.03 last 58.53) - open/close 1845 (time 1778615526.78 total 30.05 last 60.86) - open/close 2496 (time 1778615536.78 total 40.06 last 65.06) - open/close 3191 (time 1778615546.79 total 50.06 last 69.47) - open/close 3799 (time 1778615556.81 total 60.09 last 60.66) - open/close 4483 (time 1778615566.82 total 70.10 last 68.32) - open/close 5090 (time 1778615576.83 total 80.10 last 60.67) - open/close 5765 (time 1778615586.84 total 90.11 last 67.44) - open/close 6440 (time 1778615596.85 total 100.12 last 67.44) - open/close 7089 (time 1778615606.85 total 110.13 last 64.88) - open/close 7765 (time 1778615616.85 total 120.13 last 67.59) - open/close 8649 (time 1778615626.85 total 130.13 last 88.40) - open/close 9374 (time 1778615636.85 total 140.13 last 72.48) total: 10000 open/close in 148.35 seconds: 67.41 ops/second - unlinked 0 (time 1778615649 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 723 (time 1778615752.64 total 10.01 last 72.24) - open/close 1451 (time 1778615762.64 total 20.01 last 72.77) - open/close 2179 (time 1778615772.65 total 30.02 last 72.78) - open/close 2926 (time 1778615782.66 total 40.03 last 74.61) - open/close 3702 (time 1778615792.67 total 50.03 last 77.55) - open/close 4591 (time 1778615802.67 total 60.04 last 88.89) - open/close 5473 (time 1778615812.67 total 70.04 last 88.18) - open/close 6230 (time 1778615822.67 total 80.04 last 75.68) - open/close 6927 (time 1778615832.67 total 90.04 last 69.69) - open/close 7760 (time 1778615842.68 total 100.05 last 83.27) - open/close 8450 (time 1778615852.69 total 110.05 last 68.94) - open/close 9153 (time 1778615862.71 total 120.07 last 70.15) - open/close 9818 (time 1778615872.73 total 130.10 last 66.36) total: 10000 open/close in 133.18 seconds: 75.09 ops/second - unlinked 0 (time 1778615880 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 665 (time 1778615974.67 total 10.01 last 66.41) - open/close 1364 (time 1778615984.68 total 20.02 last 69.86) - open/close 2099 (time 1778615994.69 total 30.03 last 73.42) - open/close 2829 (time 1778616004.70 total 40.04 last 72.91) - open/close 3590 (time 1778616014.71 total 50.05 last 76.02) - open/close 4295 (time 1778616024.72 total 60.06 last 70.43) - open/close 5018 (time 1778616034.74 total 70.07 last 72.21) - open/close 5775 (time 1778616044.74 total 80.08 last 75.65) - open/close 6516 (time 1778616054.76 total 90.09 last 74.00) - open/close 7135 (time 1778616064.76 total 100.10 last 61.89) - open/close 7892 (time 1778616074.76 total 110.10 last 75.69) - open/close 8735 (time 1778616084.77 total 120.10 last 84.23) - open/close 9365 (time 1778616094.77 total 130.10 last 63.00) total: 10000 open/close in 137.82 seconds: 72.56 ops/second - unlinked 0 (time 1778616108 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 792 (time 1778616203.62 total 10.01 last 79.13) - open/close 1561 (time 1778616213.63 total 20.02 last 76.80) - open/close 2247 (time 1778616223.63 total 30.02 last 68.59) - open/close 2954 (time 1778616233.64 total 40.03 last 70.67) - open/close 3637 (time 1778616243.64 total 50.03 last 68.26) - open/close 4411 (time 1778616253.65 total 60.04 last 77.37) - open/close 5252 (time 1778616263.65 total 70.04 last 84.04) - open/close 5953 (time 1778616273.67 total 80.06 last 70.02) - open/close 6636 (time 1778616283.68 total 90.07 last 68.22) - open/close 7341 (time 1778616293.69 total 100.08 last 70.44) - open/close 8035 (time 1778616303.70 total 110.09 last 69.33) - open/close 8819 (time 1778616313.70 total 120.09 last 78.40) - open/close 9412 (time 1778616323.71 total 130.10 last 59.21) - open/close 9937 (time 1778616333.73 total 140.12 last 52.41) total: 10000 open/close in 141.01 seconds: 70.92 ops/second - unlinked 0 (time 1778616340 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 697 (time 1778616438.61 total 10.01 last 69.66) - open/close 1311 (time 1778616448.62 total 20.02 last 61.34) - open/close 2002 (time 1778616458.62 total 30.02 last 69.10) - open/close 2671 (time 1778616468.62 total 40.02 last 66.89) - open/close 3305 (time 1778616478.63 total 50.03 last 63.34) - open/close 3923 (time 1778616488.65 total 60.05 last 61.68) - open/close 4548 (time 1778616498.66 total 70.05 last 62.45) - open/close 5189 (time 1778616508.67 total 80.06 last 64.04) - open/close 5744 (time 1778616518.67 total 90.07 last 55.46) - open/close 6407 (time 1778616528.68 total 100.07 last 66.30) - open/close 6949 (time 1778616538.69 total 110.09 last 54.11) - open/close 7606 (time 1778616548.69 total 120.09 last 65.68) - open/close 8343 (time 1778616558.71 total 130.10 last 73.61) - open/close 9020 (time 1778616568.72 total 140.12 last 67.60) - open/close 9657 (time 1778616578.74 total 150.13 last 63.60) total: 10000 open/close in 155.19 seconds: 64.44 ops/second - unlinked 0 (time 1778616590 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.108@tcp sys.timeout=20 start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg208-server: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg208-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: modules unloaded. pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2