== conf-sanity test 69: replace an OST with the same index ========================================================== 05:56:50 (1787392610) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 784 (time 1787392734.91 total 10.01 last 78.35) - open/close 1747 (time 1787392744.92 total 20.01 last 96.24) - open/close 2833 (time 1787392754.92 total 30.02 last 108.54) - open/close 3708 (time 1787392764.93 total 40.02 last 87.44) - open/close 4614 (time 1787392774.94 total 50.04 last 90.49) - open/close 5323 (time 1787392784.95 total 60.04 last 70.85) - open/close 6109 (time 1787392794.96 total 70.06 last 78.49) - open/close 7126 (time 1787392804.97 total 80.06 last 101.62) - open/close 7918 (time 1787392814.97 total 90.07 last 79.18) - open/close 8797 (time 1787392824.97 total 100.07 last 87.90) - open/close 9535 (time 1787392834.98 total 110.07 last 73.77) total: 10000 open/close in 115.38 seconds: 86.67 ops/second - unlinked 0 (time 1787392845 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 791 (time 1787392926.96 total 10.01 last 79.04) - open/close 1637 (time 1787392936.97 total 20.02 last 84.50) - open/close 2332 (time 1787392946.97 total 30.02 last 69.49) - open/close 3099 (time 1787392956.99 total 40.04 last 76.57) - open/close 3919 (time 1787392966.99 total 50.04 last 82.00) - open/close 4823 (time 1787392976.99 total 60.04 last 90.38) - open/close 5692 (time 1787392987.00 total 70.04 last 86.87) - open/close 6527 (time 1787392997.00 total 80.05 last 83.43) - open/close 7362 (time 1787393007.03 total 90.08 last 83.28) - open/close 8074 (time 1787393017.04 total 100.09 last 71.10) - open/close 8787 (time 1787393027.06 total 110.11 last 71.21) - open/close 9475 (time 1787393037.07 total 120.11 last 68.75) total: 10000 open/close in 126.52 seconds: 79.04 ops/second - unlinked 0 (time 1787393048 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 798 (time 1787393132.67 total 10.01 last 79.75) - open/close 1521 (time 1787393142.69 total 20.02 last 72.20) - open/close 2288 (time 1787393152.70 total 30.04 last 76.59) - open/close 3107 (time 1787393162.70 total 40.04 last 81.88) - open/close 4103 (time 1787393172.71 total 50.04 last 99.53) - open/close 5122 (time 1787393182.71 total 60.05 last 101.85) - open/close 6141 (time 1787393192.71 total 70.05 last 101.89) - open/close 7171 (time 1787393202.72 total 80.06 last 102.94) - open/close 8049 (time 1787393212.73 total 90.06 last 87.74) - open/close 8911 (time 1787393222.74 total 100.08 last 86.05) - open/close 9779 (time 1787393232.75 total 110.08 last 86.78) total: 10000 open/close in 112.17 seconds: 89.15 ops/second - unlinked 0 (time 1787393239 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 1137 (time 1787393314.32 total 10.01 last 113.63) - open/close 1998 (time 1787393324.32 total 20.01 last 86.07) - open/close 3022 (time 1787393334.34 total 30.03 last 102.22) - open/close 4009 (time 1787393344.34 total 40.03 last 98.66) - open/close 4976 (time 1787393354.35 total 50.04 last 96.61) - open/close 5474 (time 1787393364.35 total 60.04 last 49.79) - open/close 6374 (time 1787393374.36 total 70.05 last 89.92) - open/close 7510 (time 1787393384.37 total 80.06 last 113.53) - open/close 8626 (time 1787393394.37 total 90.06 last 111.59) - open/close 9680 (time 1787393404.37 total 100.06 last 105.36) total: 10000 open/close in 103.46 seconds: 96.66 ops/second - unlinked 0 (time 1787393412 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 929 (time 1787393486.14 total 10.00 last 92.86) - open/close 1957 (time 1787393496.15 total 20.01 last 102.74) - open/close 2934 (time 1787393506.15 total 30.02 last 97.65) - open/close 3619 (time 1787393516.16 total 40.02 last 68.46) - open/close 4493 (time 1787393526.16 total 50.02 last 87.39) - open/close 5462 (time 1787393536.16 total 60.02 last 96.90) - open/close 6424 (time 1787393546.17 total 70.03 last 96.09) - open/close 7226 (time 1787393556.17 total 80.04 last 80.17) - open/close 7827 (time 1787393566.19 total 90.05 last 60.01) - open/close 8757 (time 1787393576.20 total 100.06 last 92.92) - open/close 9682 (time 1787393586.20 total 110.06 last 92.49) total: 10000 open/close in 113.28 seconds: 88.28 ops/second - unlinked 0 (time 1787393593 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 867 (time 1787393667.69 total 10.01 last 86.59) - open/close 1631 (time 1787393677.70 total 20.02 last 76.31) - open/close 2337 (time 1787393687.70 total 30.02 last 70.59) - open/close 3062 (time 1787393697.70 total 40.03 last 72.49) - open/close 4096 (time 1787393707.71 total 50.03 last 103.33) - open/close 5185 (time 1787393717.71 total 60.04 last 108.86) - open/close 6176 (time 1787393727.71 total 70.04 last 99.09) - open/close 7233 (time 1787393737.72 total 80.04 last 105.69) - open/close 8458 (time 1787393747.72 total 90.04 last 122.50) - open/close 9314 (time 1787393757.72 total 100.04 last 85.59) total: 10000 open/close in 107.48 seconds: 93.04 ops/second - unlinked 0 (time 1787393769 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1087 (time 1787393849.93 total 10.00 last 108.66) - open/close 2072 (time 1787393859.94 total 20.01 last 98.45) - open/close 3063 (time 1787393869.94 total 30.01 last 99.04) - open/close 4243 (time 1787393879.95 total 40.02 last 117.96) - open/close 5272 (time 1787393889.98 total 50.05 last 102.59) - open/close 6287 (time 1787393899.98 total 60.05 last 101.45) - open/close 7269 (time 1787393909.98 total 70.05 last 98.18) - open/close 8076 (time 1787393919.99 total 80.06 last 80.69) - open/close 9042 (time 1787393930.00 total 90.07 last 96.51) total: 10000 open/close in 98.87 seconds: 101.14 ops/second - unlinked 0 (time 1787393942 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1010 (time 1787394014.89 total 10.01 last 100.93) - open/close 1905 (time 1787394024.89 total 20.01 last 89.49) - open/close 2768 (time 1787394034.89 total 30.01 last 86.28) - open/close 3657 (time 1787394044.90 total 40.01 last 88.87) - open/close 4529 (time 1787394054.90 total 50.02 last 87.14) - open/close 5527 (time 1787394064.91 total 60.02 last 99.80) - open/close 6488 (time 1787394074.91 total 70.03 last 96.04) - open/close 7568 (time 1787394084.91 total 80.03 last 107.97) - open/close 8563 (time 1787394094.92 total 90.04 last 99.43) - open/close 9569 (time 1787394104.92 total 100.04 last 100.59) total: 10000 open/close in 104.40 seconds: 95.79 ops/second - unlinked 0 (time 1787394112 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 874 (time 1787394187.31 total 10.00 last 87.38) - open/close 1736 (time 1787394197.32 total 20.02 last 86.05) - open/close 2734 (time 1787394207.34 total 30.04 last 99.59) - open/close 3522 (time 1787394217.35 total 40.04 last 78.79) - open/close 4480 (time 1787394227.35 total 50.05 last 95.76) - open/close 5428 (time 1787394237.35 total 60.05 last 94.75) - open/close 6436 (time 1787394247.36 total 70.06 last 100.73) - open/close 7474 (time 1787394257.37 total 80.06 last 103.75) - open/close 8457 (time 1787394267.38 total 90.08 last 98.16) - open/close 9196 (time 1787394277.39 total 100.08 last 73.85) total: 10000 open/close in 109.16 seconds: 91.61 ops/second - unlinked 0 (time 1787394290 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 943 (time 1787394371.58 total 10.01 last 94.25) - open/close 1919 (time 1787394381.58 total 20.01 last 97.57) - open/close 2929 (time 1787394391.59 total 30.02 last 100.93) - open/close 3924 (time 1787394401.59 total 40.02 last 99.48) - open/close 4897 (time 1787394411.59 total 50.02 last 97.28) - open/close 5925 (time 1787394421.59 total 60.02 last 102.77) - open/close 6882 (time 1787394431.60 total 70.03 last 95.66) - open/close 7921 (time 1787394441.60 total 80.03 last 103.85) - open/close 8908 (time 1787394451.61 total 90.04 last 98.66) - open/close 9778 (time 1787394461.62 total 100.05 last 86.87) total: 10000 open/close in 102.18 seconds: 97.86 ops/second - unlinked 0 (time 1787394467 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.104@tcp sys.timeout=20 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2