== conf-sanity test 69: replace an OST with the same index ========================================================== 10:48:41 (1788792521) start mds service on oleg113-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg113-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg113-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 508 (time 1788792655.08 total 10.01 last 50.77) - open/close 1161 (time 1788792665.09 total 20.02 last 65.24) - open/close 1932 (time 1788792675.10 total 30.03 last 76.99) - open/close 2687 (time 1788792685.11 total 40.03 last 75.46) - open/close 3355 (time 1788792695.12 total 50.04 last 66.76) - open/close 4042 (time 1788792705.12 total 60.05 last 68.65) - open/close 4657 (time 1788792715.12 total 70.05 last 61.49) - open/close 5495 (time 1788792725.13 total 80.06 last 83.73) - open/close 6110 (time 1788792735.14 total 90.06 last 61.45) - open/close 6784 (time 1788792745.15 total 100.07 last 67.34) - open/close 7387 (time 1788792755.16 total 110.08 last 60.24) - open/close 7828 (time 1788792765.16 total 120.09 last 44.09) - open/close 8379 (time 1788792775.17 total 130.09 last 55.07) - open/close 9097 (time 1788792785.17 total 140.10 last 71.77) - open/close 9760 (time 1788792795.18 total 150.10 last 66.26) total: 10000 open/close in 153.63 seconds: 65.09 ops/second - unlinked 0 (time 1788792805 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 719 (time 1788792911.29 total 10.00 last 71.88) - open/close 1568 (time 1788792921.29 total 20.00 last 84.90) - open/close 2338 (time 1788792931.30 total 30.01 last 76.96) - open/close 2998 (time 1788792941.32 total 40.03 last 65.86) - open/close 3498 (time 1788792951.35 total 50.07 last 49.83) - open/close 4088 (time 1788792961.36 total 60.08 last 58.93) - open/close 4840 (time 1788792971.38 total 70.10 last 75.04) - open/close 5385 (time 1788792981.39 total 80.11 last 54.45) - open/close 5996 (time 1788792991.42 total 90.13 last 60.96) - open/close 6718 (time 1788793001.42 total 100.13 last 72.19) - open/close 7488 (time 1788793011.43 total 110.14 last 76.92) - open/close 8252 (time 1788793021.44 total 120.15 last 76.31) - open/close 8985 (time 1788793031.44 total 130.16 last 73.29) - open/close 9779 (time 1788793041.46 total 140.18 last 79.24) total: 10000 open/close in 143.30 seconds: 69.78 ops/second - unlinked 0 (time 1788793049 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 918 (time 1788793133.91 total 10.01 last 91.71) - open/close 1549 (time 1788793143.93 total 20.03 last 62.97) - open/close 2254 (time 1788793153.95 total 30.06 last 70.33) - open/close 2962 (time 1788793163.96 total 40.06 last 70.77) - open/close 3861 (time 1788793173.97 total 50.07 last 89.81) - open/close 4686 (time 1788793183.97 total 60.08 last 82.45) - open/close 5462 (time 1788793193.98 total 70.08 last 77.57) - open/close 6302 (time 1788793203.98 total 80.09 last 83.95) - open/close 7050 (time 1788793213.98 total 90.09 last 74.80) - open/close 7723 (time 1788793223.99 total 100.09 last 67.26) - open/close 8392 (time 1788793234.00 total 110.10 last 66.83) - open/close 9127 (time 1788793244.02 total 120.13 last 73.33) - open/close 9876 (time 1788793254.02 total 130.13 last 74.90) total: 10000 open/close in 131.44 seconds: 76.08 ops/second - unlinked 0 (time 1788793260 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 851 (time 1788793341.49 total 10.01 last 85.05) - open/close 1913 (time 1788793351.49 total 20.01 last 106.14) - open/close 2943 (time 1788793361.50 total 30.02 last 102.95) - open/close 3819 (time 1788793371.50 total 40.02 last 87.56) - open/close 4588 (time 1788793381.50 total 50.02 last 76.88) - open/close 5691 (time 1788793391.51 total 60.03 last 110.24) - open/close 6501 (time 1788793401.51 total 70.03 last 80.97) - open/close 7491 (time 1788793411.52 total 80.04 last 98.96) - open/close 8557 (time 1788793421.52 total 90.04 last 106.58) - open/close 9585 (time 1788793431.52 total 100.04 last 102.76) total: 10000 open/close in 104.00 seconds: 96.16 ops/second - unlinked 0 (time 1788793439 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 767 (time 1788793516.86 total 10.02 last 76.58) - open/close 1819 (time 1788793526.87 total 20.02 last 105.15) - open/close 3008 (time 1788793536.87 total 30.03 last 118.82) - open/close 4118 (time 1788793546.88 total 40.03 last 110.94) - open/close 5252 (time 1788793556.88 total 50.04 last 113.36) - open/close 6230 (time 1788793566.89 total 60.05 last 97.70) - open/close 7070 (time 1788793576.90 total 70.05 last 83.96) - open/close 8122 (time 1788793586.91 total 80.06 last 105.12) - open/close 9209 (time 1788793596.91 total 90.06 last 108.67) total: 10000 open/close in 97.81 seconds: 102.24 ops/second - unlinked 0 (time 1788793608 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1079 (time 1788793688.61 total 10.01 last 107.84) - open/close 2074 (time 1788793698.62 total 20.01 last 99.49) - open/close 3158 (time 1788793708.62 total 30.01 last 108.37) - open/close 4230 (time 1788793718.63 total 40.02 last 107.11) - open/close 5320 (time 1788793728.63 total 50.03 last 108.92) - open/close 6180 (time 1788793738.64 total 60.03 last 85.94) - open/close 7321 (time 1788793748.65 total 70.04 last 114.05) - open/close 8400 (time 1788793758.65 total 80.04 last 107.85) - open/close 9705 (time 1788793768.66 total 90.05 last 130.39) total: 10000 open/close in 92.41 seconds: 108.21 ops/second - unlinked 0 (time 1788793775 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 690 (time 1788793856.17 total 10.00 last 68.97) - open/close 1524 (time 1788793866.17 total 20.01 last 83.37) - open/close 2418 (time 1788793876.18 total 30.01 last 89.38) - open/close 3366 (time 1788793886.18 total 40.02 last 94.75) - open/close 4272 (time 1788793896.20 total 50.04 last 90.39) - open/close 5388 (time 1788793906.20 total 60.04 last 111.59) - open/close 6472 (time 1788793916.21 total 70.04 last 108.40) - open/close 7697 (time 1788793926.21 total 80.04 last 122.49) - open/close 8629 (time 1788793936.21 total 90.05 last 93.14) - open/close 9734 (time 1788793946.22 total 100.05 last 110.44) total: 10000 open/close in 102.42 seconds: 97.64 ops/second - unlinked 0 (time 1788793953 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 863 (time 1788794022.56 total 10.00 last 86.26) - open/close 1636 (time 1788794032.56 total 20.01 last 77.28) - open/close 2477 (time 1788794042.56 total 30.01 last 84.08) - open/close 3485 (time 1788794052.57 total 40.02 last 100.75) - open/close 4266 (time 1788794062.58 total 50.02 last 78.04) - open/close 5067 (time 1788794072.59 total 60.04 last 80.00) - open/close 5984 (time 1788794082.60 total 70.04 last 91.66) - open/close 6739 (time 1788794092.61 total 80.05 last 75.41) - open/close 7411 (time 1788794102.61 total 90.06 last 67.16) - open/close 8198 (time 1788794112.62 total 100.06 last 78.67) - open/close 8966 (time 1788794122.63 total 110.08 last 76.68) - open/close 9751 (time 1788794132.64 total 120.08 last 78.47) total: 10000 open/close in 122.95 seconds: 81.34 ops/second - unlinked 0 (time 1788794141 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 582 (time 1788794209.94 total 10.00 last 58.19) - open/close 1341 (time 1788794219.94 total 20.00 last 75.88) - open/close 2377 (time 1788794229.94 total 30.01 last 103.57) - open/close 3262 (time 1788794239.94 total 40.01 last 88.48) - open/close 3912 (time 1788794249.96 total 50.03 last 64.87) - open/close 4738 (time 1788794259.96 total 60.03 last 82.59) - open/close 5815 (time 1788794269.96 total 70.03 last 107.69) - open/close 6713 (time 1788794279.97 total 80.04 last 89.75) - open/close 7638 (time 1788794289.98 total 90.05 last 92.38) - open/close 8618 (time 1788794299.98 total 100.05 last 97.98) - open/close 9610 (time 1788794309.99 total 110.05 last 99.18) total: 10000 open/close in 113.50 seconds: 88.11 ops/second - unlinked 0 (time 1788794318 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1276 (time 1788794383.20 total 10.01 last 127.52) - open/close 2499 (time 1788794393.20 total 20.01 last 122.28) - open/close 3715 (time 1788794403.20 total 30.01 last 121.58) - open/close 4893 (time 1788794413.21 total 40.01 last 117.74) - open/close 5975 (time 1788794423.21 total 50.02 last 108.18) - open/close 7162 (time 1788794433.22 total 60.03 last 118.58) - open/close 8318 (time 1788794443.22 total 70.03 last 115.56) - open/close 9082 (time 1788794453.23 total 80.04 last 76.31) total: 10000 open/close in 89.38 seconds: 111.88 ops/second - unlinked 0 (time 1788794466 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg113-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.113@tcp sys.timeout=20 start ost1 service on oleg113-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg113-server: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg113-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2