== conf-sanity test 69: replace an OST with the same index ========================================================== 08:35:01 (1787747701) start mds service on oleg610-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing load_modules_local oleg610-server: Loading modules from /home/green/git/lustre-release/lustre oleg610-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg610-server: detected 4 online CPUs by sysfs oleg610-server: MODOPTS_LIBCFS= oleg610-server: Force libcfs to create 2 CPU partitions oleg610-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg610-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg610-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg610-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg610-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 737 (time 1787747814.33 total 10.02 last 73.56) - open/close 1466 (time 1787747824.34 total 20.03 last 72.81) - open/close 2190 (time 1787747834.34 total 30.03 last 72.39) - open/close 3017 (time 1787747844.34 total 40.04 last 82.67) - open/close 3798 (time 1787747854.35 total 50.04 last 78.06) - open/close 4573 (time 1787747864.35 total 60.04 last 77.47) - open/close 5331 (time 1787747874.36 total 70.05 last 75.76) - open/close 6140 (time 1787747884.36 total 80.05 last 80.89) - open/close 6778 (time 1787747894.37 total 90.07 last 63.70) - open/close 7283 (time 1787747904.38 total 100.07 last 50.47) - open/close 7930 (time 1787747914.39 total 110.08 last 64.65) - open/close 8558 (time 1787747924.40 total 120.09 last 62.74) - open/close 9211 (time 1787747934.41 total 130.10 last 65.24) - open/close 9999 (time 1787747944.41 total 140.10 last 78.78) total: 10000 open/close in 140.10 seconds: 71.38 ops/second - unlinked 0 (time 1787747948 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 787 (time 1787748037.69 total 10.01 last 78.64) - open/close 1631 (time 1787748047.69 total 20.01 last 84.39) - open/close 2320 (time 1787748057.70 total 30.01 last 68.85) - open/close 3034 (time 1787748067.71 total 40.02 last 71.34) - open/close 3731 (time 1787748077.72 total 50.04 last 69.61) - open/close 4428 (time 1787748087.72 total 60.04 last 69.69) - open/close 5247 (time 1787748097.72 total 70.04 last 81.87) - open/close 5921 (time 1787748107.72 total 80.04 last 67.40) - open/close 6699 (time 1787748117.73 total 90.04 last 77.80) - open/close 7501 (time 1787748127.73 total 100.04 last 80.18) - open/close 8331 (time 1787748137.74 total 110.05 last 82.91) - open/close 9116 (time 1787748147.74 total 120.06 last 78.49) - open/close 9844 (time 1787748157.75 total 130.06 last 72.74) total: 10000 open/close in 132.38 seconds: 75.54 ops/second - unlinked 0 (time 1787748164 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 673 (time 1787748259.02 total 10.01 last 67.24) - open/close 1458 (time 1787748269.02 total 20.01 last 78.45) - open/close 2203 (time 1787748279.04 total 30.03 last 74.42) - open/close 2964 (time 1787748289.04 total 40.03 last 76.05) - open/close 3739 (time 1787748299.05 total 50.04 last 77.46) - open/close 4465 (time 1787748309.05 total 60.04 last 72.60) - open/close 5277 (time 1787748319.05 total 70.04 last 81.15) - open/close 6071 (time 1787748329.07 total 80.06 last 79.30) - open/close 6776 (time 1787748339.07 total 90.06 last 70.45) - open/close 7627 (time 1787748349.08 total 100.07 last 85.04) - open/close 8426 (time 1787748359.09 total 110.08 last 79.82) - open/close 9179 (time 1787748369.10 total 120.09 last 75.23) - open/close 9917 (time 1787748379.12 total 130.11 last 73.67) total: 10000 open/close in 131.10 seconds: 76.28 ops/second - unlinked 0 (time 1787748383 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 727 (time 1787748473.06 total 10.02 last 72.59) - open/close 1317 (time 1787748483.07 total 20.03 last 58.91) - open/close 1921 (time 1787748493.08 total 30.04 last 60.36) - open/close 2642 (time 1787748503.08 total 40.04 last 72.08) - open/close 3372 (time 1787748513.10 total 50.06 last 72.86) - open/close 4022 (time 1787748523.11 total 60.07 last 64.95) - open/close 4696 (time 1787748533.11 total 70.07 last 67.38) - open/close 5348 (time 1787748543.11 total 80.07 last 65.20) - open/close 5990 (time 1787748553.13 total 90.08 last 64.11) - open/close 6619 (time 1787748563.13 total 100.09 last 62.86) - open/close 7305 (time 1787748573.14 total 110.10 last 68.56) - open/close 8039 (time 1787748583.15 total 120.11 last 73.30) - open/close 8822 (time 1787748593.16 total 130.12 last 78.21) - open/close 9530 (time 1787748603.17 total 140.12 last 70.79) total: 10000 open/close in 145.46 seconds: 68.75 ops/second - unlinked 0 (time 1787748613 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 837 (time 1787748697.92 total 10.02 last 83.56) - open/close 1721 (time 1787748707.92 total 20.02 last 88.39) - open/close 2523 (time 1787748717.93 total 30.03 last 80.14) - open/close 3387 (time 1787748727.93 total 40.03 last 86.34) - open/close 4196 (time 1787748737.94 total 50.03 last 80.89) - open/close 5022 (time 1787748747.94 total 60.04 last 82.57) - open/close 5802 (time 1787748757.95 total 70.05 last 77.90) - open/close 6580 (time 1787748767.96 total 80.06 last 77.76) - open/close 7354 (time 1787748777.96 total 90.06 last 77.34) - open/close 8158 (time 1787748787.98 total 100.07 last 80.31) - open/close 8923 (time 1787748797.98 total 110.08 last 76.47) - open/close 9738 (time 1787748807.99 total 120.09 last 81.45) total: 10000 open/close in 123.62 seconds: 80.89 ops/second - unlinked 0 (time 1787748817 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 828 (time 1787748903.63 total 10.00 last 82.78) - open/close 1612 (time 1787748913.64 total 20.01 last 78.34) - open/close 2347 (time 1787748923.64 total 30.02 last 73.44) - open/close 3061 (time 1787748933.64 total 40.02 last 71.40) - open/close 3852 (time 1787748943.65 total 50.03 last 79.02) - open/close 4633 (time 1787748953.66 total 60.03 last 78.09) - open/close 5369 (time 1787748963.66 total 70.03 last 73.58) - open/close 6006 (time 1787748973.67 total 80.04 last 63.64) - open/close 6751 (time 1787748983.67 total 90.05 last 74.45) - open/close 7632 (time 1787748993.69 total 100.06 last 87.99) - open/close 8444 (time 1787749003.71 total 110.09 last 81.00) - open/close 9217 (time 1787749013.73 total 120.10 last 77.19) - open/close 9946 (time 1787749023.73 total 130.11 last 72.84) total: 10000 open/close in 130.88 seconds: 76.40 ops/second - unlinked 0 (time 1787749028 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 765 (time 1787749114.88 total 10.01 last 76.46) - open/close 1476 (time 1787749124.89 total 20.02 last 71.01) - open/close 2245 (time 1787749134.90 total 30.02 last 76.86) - open/close 3063 (time 1787749144.90 total 40.02 last 81.80) - open/close 3805 (time 1787749154.90 total 50.03 last 74.18) - open/close 4606 (time 1787749164.90 total 60.03 last 80.08) - open/close 5477 (time 1787749174.90 total 70.03 last 87.07) - open/close 6305 (time 1787749184.91 total 80.04 last 82.73) - open/close 7029 (time 1787749194.93 total 90.05 last 72.31) - open/close 7842 (time 1787749204.94 total 100.07 last 81.17) - open/close 8611 (time 1787749214.95 total 110.07 last 76.85) - open/close 9493 (time 1787749224.95 total 120.08 last 88.19) total: 10000 open/close in 126.65 seconds: 78.96 ops/second - unlinked 0 (time 1787749236 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 819 (time 1787749319.08 total 10.00 last 81.89) - open/close 1612 (time 1787749329.10 total 20.02 last 79.17) - open/close 2336 (time 1787749339.11 total 30.02 last 72.35) - open/close 3104 (time 1787749349.12 total 40.04 last 76.72) - open/close 3852 (time 1787749359.14 total 50.05 last 74.66) - open/close 4448 (time 1787749369.14 total 60.05 last 59.60) - open/close 5174 (time 1787749379.14 total 70.06 last 72.59) - open/close 5928 (time 1787749389.14 total 80.06 last 75.35) - open/close 6643 (time 1787749399.15 total 90.07 last 71.46) - open/close 7344 (time 1787749409.15 total 100.07 last 70.10) - open/close 8102 (time 1787749419.16 total 110.07 last 75.76) - open/close 8838 (time 1787749429.16 total 120.08 last 73.59) - open/close 9571 (time 1787749439.16 total 130.08 last 73.30) total: 10000 open/close in 135.39 seconds: 73.86 ops/second - unlinked 0 (time 1787749449 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 787 (time 1787749536.50 total 10.01 last 78.62) - open/close 1525 (time 1787749546.50 total 20.01 last 73.78) - open/close 2339 (time 1787749556.51 total 30.03 last 81.30) - open/close 3062 (time 1787749566.52 total 40.03 last 72.24) - open/close 3779 (time 1787749576.53 total 50.04 last 71.65) - open/close 4435 (time 1787749586.53 total 60.05 last 65.57) - open/close 5121 (time 1787749596.57 total 70.08 last 68.36) - open/close 5758 (time 1787749606.58 total 80.09 last 63.63) - open/close 6507 (time 1787749616.58 total 90.09 last 74.89) - open/close 7132 (time 1787749626.59 total 100.10 last 62.46) - open/close 7944 (time 1787749636.59 total 110.10 last 81.16) - open/close 8771 (time 1787749646.60 total 120.11 last 82.63) - open/close 9521 (time 1787749656.61 total 130.13 last 74.89) total: 10000 open/close in 136.66 seconds: 73.18 ops/second - unlinked 0 (time 1787749668 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 613 (time 1787749755.89 total 10.01 last 61.27) - open/close 1362 (time 1787749765.90 total 20.01 last 74.88) - open/close 2032 (time 1787749775.90 total 30.01 last 66.96) - open/close 2888 (time 1787749785.90 total 40.02 last 85.60) - open/close 3710 (time 1787749795.91 total 50.02 last 82.12) - open/close 4402 (time 1787749805.93 total 60.04 last 69.06) - open/close 5164 (time 1787749815.94 total 70.05 last 76.15) - open/close 5896 (time 1787749825.95 total 80.06 last 73.11) - open/close 6586 (time 1787749835.95 total 90.07 last 68.98) - open/close 7246 (time 1787749845.96 total 100.08 last 65.94) - open/close 7996 (time 1787749855.97 total 110.08 last 74.96) - open/close 8643 (time 1787749865.98 total 120.09 last 64.63) - open/close 9440 (time 1787749875.99 total 130.11 last 79.57) total: 10000 open/close in 137.24 seconds: 72.86 ops/second - unlinked 0 (time 1787749887 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg610-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg610-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg610-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg610-server: oleg610-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg610-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg610-server: oleg610-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg610-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg610-client: oleg610-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg610-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg610-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.110@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.110@tcp sys.timeout=20 start ost1 service on oleg610-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg610-server: oleg610-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg610-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg610-server: oleg610-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg610-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg610-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg610-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server stop mds service on oleg610-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg610-server stop mds service on oleg610-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg610-server unloading modules via unload_modules_local on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing unload_modules_local oleg610-server: modules unloaded. pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2