== conf-sanity test 69: replace an OST with the same index ========================================================== 16:34:11 (1787862851) start mds service on oleg153-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg153-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg153-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49502 inodes available. Want 99872. rc=0 - open/close 873 (time 1787862963.44 total 10.00 last 87.27) - open/close 1624 (time 1787862973.44 total 20.01 last 75.05) - open/close 2357 (time 1787862983.45 total 30.02 last 73.23) - open/close 3164 (time 1787862993.46 total 40.02 last 80.66) - open/close 3936 (time 1787863003.46 total 50.02 last 77.20) - open/close 4565 (time 1787863013.47 total 60.03 last 62.86) - open/close 5343 (time 1787863023.47 total 70.04 last 77.75) - open/close 6044 (time 1787863033.48 total 80.05 last 70.04) - open/close 6892 (time 1787863043.49 total 90.05 last 84.76) - open/close 7706 (time 1787863053.49 total 100.05 last 81.38) - open/close 8317 (time 1787863063.50 total 110.06 last 61.04) - open/close 8891 (time 1787863073.50 total 120.07 last 57.37) - open/close 9536 (time 1787863083.51 total 130.08 last 64.43) total: 10000 open/close in 137.43 seconds: 72.77 ops/second - unlinked 0 (time 1787863096 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 664 (time 1787863190.07 total 10.02 last 66.28) - open/close 1259 (time 1787863200.07 total 20.02 last 59.49) - open/close 1904 (time 1787863210.08 total 30.02 last 64.47) - open/close 2626 (time 1787863220.08 total 40.03 last 72.20) - open/close 3401 (time 1787863230.10 total 50.04 last 77.37) - open/close 4285 (time 1787863240.10 total 60.04 last 88.38) - open/close 5129 (time 1787863250.11 total 70.06 last 84.29) - open/close 5729 (time 1787863260.12 total 80.07 last 59.92) - open/close 6463 (time 1787863270.13 total 90.08 last 73.36) - open/close 7249 (time 1787863280.14 total 100.08 last 78.55) - open/close 7905 (time 1787863290.14 total 110.09 last 65.54) - open/close 8621 (time 1787863300.16 total 120.11 last 71.47) - open/close 9343 (time 1787863310.17 total 130.12 last 72.13) - open/close 9931 (time 1787863320.17 total 140.12 last 58.78) total: 10000 open/close in 141.27 seconds: 70.79 ops/second - unlinked 0 (time 1787863327 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 791 (time 1787863420.87 total 10.01 last 79.04) - open/close 1439 (time 1787863430.88 total 20.02 last 64.72) - open/close 2173 (time 1787863440.89 total 30.04 last 73.27) - open/close 2897 (time 1787863450.89 total 40.04 last 72.40) - open/close 3542 (time 1787863460.91 total 50.05 last 64.43) - open/close 4288 (time 1787863470.91 total 60.05 last 74.60) - open/close 5064 (time 1787863480.91 total 70.05 last 77.56) - open/close 5933 (time 1787863490.91 total 80.05 last 86.89) - open/close 6730 (time 1787863500.92 total 90.06 last 79.61) - open/close 7495 (time 1787863510.94 total 100.08 last 76.38) - open/close 8282 (time 1787863520.95 total 110.09 last 78.62) - open/close 9010 (time 1787863530.96 total 120.10 last 72.72) - open/close 9828 (time 1787863540.96 total 130.10 last 81.79) total: 10000 open/close in 132.39 seconds: 75.53 ops/second - unlinked 0 (time 1787863548 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 685 (time 1787863647.60 total 10.01 last 68.44) - open/close 1295 (time 1787863657.62 total 20.03 last 60.88) - open/close 1922 (time 1787863667.63 total 30.04 last 62.61) - open/close 2616 (time 1787863677.63 total 40.05 last 69.38) - open/close 3300 (time 1787863687.65 total 50.06 last 68.30) - open/close 3985 (time 1787863697.66 total 60.07 last 68.45) - open/close 4682 (time 1787863707.66 total 70.07 last 69.66) - open/close 5440 (time 1787863717.66 total 80.08 last 75.78) - open/close 6232 (time 1787863727.67 total 90.08 last 79.16) - open/close 6978 (time 1787863737.67 total 100.09 last 74.56) - open/close 7771 (time 1787863747.68 total 110.09 last 79.25) - open/close 8560 (time 1787863757.69 total 120.10 last 78.82) - open/close 9256 (time 1787863767.69 total 130.10 last 69.58) - open/close 9961 (time 1787863777.70 total 140.11 last 70.44) total: 10000 open/close in 140.54 seconds: 71.15 ops/second - unlinked 0 (time 1787863783 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 696 (time 1787863876.82 total 10.00 last 69.57) - open/close 1446 (time 1787863886.84 total 20.02 last 74.89) - open/close 2256 (time 1787863896.85 total 30.03 last 80.92) - open/close 3094 (time 1787863906.86 total 40.04 last 83.73) - open/close 3854 (time 1787863916.87 total 50.05 last 75.93) - open/close 4740 (time 1787863926.87 total 60.05 last 88.54) - open/close 5624 (time 1787863936.88 total 70.06 last 88.35) - open/close 6417 (time 1787863946.89 total 80.07 last 79.22) - open/close 7214 (time 1787863956.89 total 90.08 last 79.65) - open/close 8009 (time 1787863966.90 total 100.08 last 79.44) - open/close 8752 (time 1787863976.91 total 110.09 last 74.26) - open/close 9434 (time 1787863986.91 total 120.09 last 68.19) total: 10000 open/close in 126.66 seconds: 78.95 ops/second - unlinked 0 (time 1787863998 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 631 (time 1787864086.40 total 10.00 last 63.08) - open/close 1333 (time 1787864096.40 total 20.01 last 70.17) - open/close 2177 (time 1787864106.41 total 30.01 last 84.35) - open/close 3008 (time 1787864116.42 total 40.03 last 82.97) - open/close 3869 (time 1787864126.43 total 50.04 last 86.04) - open/close 4754 (time 1787864136.43 total 60.04 last 88.47) - open/close 5462 (time 1787864146.44 total 70.05 last 70.73) - open/close 6169 (time 1787864156.45 total 80.06 last 70.65) - open/close 7041 (time 1787864166.46 total 90.07 last 87.11) - open/close 7750 (time 1787864176.50 total 100.10 last 70.66) - open/close 8365 (time 1787864186.51 total 110.11 last 61.43) - open/close 9015 (time 1787864196.52 total 120.12 last 64.93) - open/close 9761 (time 1787864206.53 total 130.13 last 74.52) total: 10000 open/close in 133.25 seconds: 75.05 ops/second - unlinked 0 (time 1787864214 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 686 (time 1787864305.62 total 10.00 last 68.59) - open/close 1332 (time 1787864315.63 total 20.01 last 64.53) - open/close 2094 (time 1787864325.63 total 30.02 last 76.15) - open/close 2870 (time 1787864335.67 total 40.06 last 77.30) - open/close 3578 (time 1787864345.68 total 50.06 last 70.77) - open/close 4334 (time 1787864355.68 total 60.06 last 75.58) - open/close 5209 (time 1787864365.71 total 70.09 last 87.26) - open/close 6012 (time 1787864375.72 total 80.11 last 80.18) - open/close 6769 (time 1787864385.72 total 90.11 last 75.68) - open/close 7531 (time 1787864395.73 total 100.12 last 76.14) - open/close 8356 (time 1787864405.74 total 110.13 last 82.40) - open/close 9128 (time 1787864415.75 total 120.13 last 77.18) total: 10000 open/close in 130.07 seconds: 76.88 ops/second - unlinked 0 (time 1787864429 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 731 (time 1787864515.66 total 10.00 last 73.08) - open/close 1597 (time 1787864525.67 total 20.01 last 86.51) - open/close 2348 (time 1787864535.69 total 30.02 last 75.02) - open/close 3066 (time 1787864545.69 total 40.03 last 71.76) - open/close 3980 (time 1787864555.70 total 50.03 last 91.36) - open/close 4909 (time 1787864565.70 total 60.04 last 92.88) - open/close 5807 (time 1787864575.70 total 70.04 last 89.76) - open/close 6477 (time 1787864585.73 total 80.07 last 66.81) - open/close 7278 (time 1787864595.74 total 90.07 last 80.05) - open/close 8090 (time 1787864605.75 total 100.09 last 81.11) - open/close 8936 (time 1787864615.76 total 110.09 last 84.54) - open/close 9753 (time 1787864625.77 total 120.10 last 81.61) total: 10000 open/close in 123.13 seconds: 81.21 ops/second - unlinked 0 (time 1787864633 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 764 (time 1787864723.64 total 10.01 last 76.34) - open/close 1597 (time 1787864733.65 total 20.02 last 83.20) - open/close 2554 (time 1787864743.66 total 30.02 last 95.67) - open/close 3431 (time 1787864753.66 total 40.03 last 87.64) - open/close 4274 (time 1787864763.67 total 50.03 last 84.28) - open/close 5078 (time 1787864773.67 total 60.04 last 80.34) - open/close 5956 (time 1787864783.68 total 70.04 last 87.78) - open/close 6821 (time 1787864793.68 total 80.04 last 86.48) - open/close 7697 (time 1787864803.68 total 90.05 last 87.56) - open/close 8566 (time 1787864813.69 total 100.05 last 86.87) - open/close 9335 (time 1787864823.69 total 110.06 last 76.83) total: 10000 open/close in 117.35 seconds: 85.21 ops/second - unlinked 0 (time 1787864836 ; total 1 ; last 1) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 757 (time 1787864921.64 total 10.01 last 75.65) - open/close 1537 (time 1787864931.64 total 20.01 last 78.00) - open/close 2211 (time 1787864941.64 total 30.01 last 67.38) - open/close 2966 (time 1787864951.66 total 40.03 last 75.37) - open/close 3592 (time 1787864961.66 total 50.03 last 62.58) - open/close 4396 (time 1787864971.67 total 60.04 last 80.35) - open/close 5194 (time 1787864981.68 total 70.05 last 79.69) - open/close 6034 (time 1787864991.70 total 80.06 last 83.89) - open/close 6855 (time 1787865001.70 total 90.06 last 82.08) - open/close 7648 (time 1787865011.70 total 100.07 last 79.29) - open/close 8428 (time 1787865021.71 total 110.07 last 77.95) - open/close 9204 (time 1787865031.71 total 120.07 last 77.60) total: 10000 open/close in 128.56 seconds: 77.78 ops/second - unlinked 0 (time 1787865044 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre On OST0, 20431 used inodes rc=0 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local oleg153-server: modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2