== conf-sanity test 69: replace an OST with the same index ========================================================== 21:36:19 (1787708179) start mds service on oleg242-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg242-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 702 (time 1787708309.46 total 10.00 last 70.19) - open/close 1394 (time 1787708319.46 total 20.00 last 69.18) - open/close 2171 (time 1787708329.47 total 30.01 last 77.67) - open/close 2813 (time 1787708339.47 total 40.01 last 64.20) - open/close 3478 (time 1787708349.48 total 50.02 last 66.43) - open/close 4127 (time 1787708359.49 total 60.04 last 64.79) - open/close 4856 (time 1787708369.50 total 70.04 last 72.90) - open/close 5572 (time 1787708379.50 total 80.04 last 71.56) - open/close 6303 (time 1787708389.52 total 90.06 last 72.99) - open/close 7026 (time 1787708399.52 total 100.06 last 72.26) - open/close 7841 (time 1787708409.53 total 110.07 last 81.44) - open/close 8754 (time 1787708419.53 total 120.08 last 91.26) - open/close 9380 (time 1787708429.54 total 130.08 last 62.58) total: 10000 open/close in 139.26 seconds: 71.81 ops/second - unlinked 0 (time 1787708443 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 712 (time 1787708532.72 total 10.01 last 71.10) - open/close 1339 (time 1787708542.72 total 20.02 last 62.67) - open/close 2171 (time 1787708552.72 total 30.02 last 83.19) - open/close 2939 (time 1787708562.73 total 40.02 last 76.78) - open/close 3678 (time 1787708572.73 total 50.03 last 73.87) - open/close 4487 (time 1787708582.74 total 60.03 last 80.84) - open/close 5180 (time 1787708592.75 total 70.05 last 69.22) - open/close 5719 (time 1787708602.75 total 80.05 last 53.90) - open/close 6194 (time 1787708612.76 total 90.05 last 47.46) - open/close 7041 (time 1787708622.77 total 100.07 last 84.59) - open/close 7797 (time 1787708632.78 total 110.07 last 75.56) - open/close 8640 (time 1787708642.79 total 120.08 last 84.19) - open/close 9494 (time 1787708652.80 total 130.09 last 85.33) total: 10000 open/close in 135.70 seconds: 73.69 ops/second - unlinked 0 (time 1787708663 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 645 (time 1787708748.00 total 10.01 last 64.41) - open/close 1355 (time 1787708758.01 total 20.02 last 70.93) - open/close 2121 (time 1787708768.01 total 30.03 last 76.55) - open/close 2896 (time 1787708778.02 total 40.03 last 77.48) - open/close 3527 (time 1787708788.04 total 50.06 last 62.96) - open/close 4097 (time 1787708798.05 total 60.06 last 56.96) - open/close 4860 (time 1787708808.05 total 70.07 last 76.28) - open/close 5659 (time 1787708818.06 total 80.07 last 79.84) - open/close 6504 (time 1787708828.06 total 90.08 last 84.49) - open/close 7264 (time 1787708838.06 total 100.08 last 75.99) - open/close 8036 (time 1787708848.06 total 110.08 last 77.16) - open/close 8893 (time 1787708858.07 total 120.09 last 85.63) - open/close 9716 (time 1787708868.08 total 130.10 last 82.21) total: 10000 open/close in 133.60 seconds: 74.85 ops/second - unlinked 0 (time 1787708876 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 714 (time 1787708960.97 total 10.00 last 71.39) - open/close 1528 (time 1787708970.98 total 20.01 last 81.34) - open/close 2251 (time 1787708980.98 total 30.01 last 72.29) - open/close 3035 (time 1787708990.99 total 40.02 last 78.33) - open/close 3664 (time 1787709000.99 total 50.03 last 62.85) - open/close 4265 (time 1787709011.01 total 60.04 last 60.01) - open/close 4975 (time 1787709021.05 total 70.08 last 70.73) - open/close 5654 (time 1787709031.05 total 80.09 last 67.85) - open/close 6221 (time 1787709041.08 total 90.11 last 56.58) - open/close 6848 (time 1787709051.10 total 100.13 last 62.56) - open/close 7638 (time 1787709061.10 total 110.13 last 78.98) - open/close 8498 (time 1787709071.11 total 120.14 last 85.96) - open/close 9226 (time 1787709081.11 total 130.14 last 72.77) - open/close 9957 (time 1787709091.11 total 140.15 last 73.07) total: 10000 open/close in 140.54 seconds: 71.15 ops/second - unlinked 0 (time 1787709096 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 740 (time 1787709186.49 total 10.00 last 73.99) - open/close 1491 (time 1787709196.49 total 20.00 last 75.08) - open/close 2270 (time 1787709206.50 total 30.02 last 77.80) - open/close 2991 (time 1787709216.50 total 40.02 last 72.10) - open/close 3735 (time 1787709226.51 total 50.02 last 74.34) - open/close 4453 (time 1787709236.52 total 60.04 last 71.70) - open/close 5021 (time 1787709246.54 total 70.06 last 56.69) - open/close 5835 (time 1787709256.55 total 80.07 last 81.31) - open/close 6623 (time 1787709266.57 total 90.08 last 78.71) - open/close 7254 (time 1787709276.58 total 100.10 last 63.02) - open/close 7930 (time 1787709286.60 total 110.11 last 67.49) - open/close 8632 (time 1787709296.61 total 120.12 last 70.13) - open/close 9415 (time 1787709306.62 total 130.13 last 78.20) total: 10000 open/close in 138.93 seconds: 71.98 ops/second - unlinked 0 (time 1787709320 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 714 (time 1787709409.99 total 10.00 last 71.38) - open/close 1454 (time 1787709420.00 total 20.01 last 73.95) - open/close 2264 (time 1787709430.00 total 30.01 last 80.98) - open/close 3035 (time 1787709440.01 total 40.02 last 77.07) - open/close 3844 (time 1787709450.01 total 50.02 last 80.86) - open/close 4703 (time 1787709460.01 total 60.02 last 85.89) - open/close 5550 (time 1787709470.02 total 70.03 last 84.66) - open/close 6419 (time 1787709480.02 total 80.03 last 86.89) - open/close 7045 (time 1787709490.02 total 90.03 last 62.59) - open/close 7865 (time 1787709500.03 total 100.04 last 81.95) - open/close 8705 (time 1787709510.04 total 110.05 last 83.91) - open/close 9542 (time 1787709520.04 total 120.05 last 83.68) total: 10000 open/close in 126.52 seconds: 79.04 ops/second - unlinked 0 (time 1787709531 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 720 (time 1787709626.99 total 10.01 last 71.96) - open/close 1501 (time 1787709636.99 total 20.01 last 78.08) - open/close 2192 (time 1787709647.00 total 30.01 last 69.07) - open/close 2868 (time 1787709657.04 total 40.05 last 67.33) - open/close 3336 (time 1787709667.04 total 50.06 last 46.80) - open/close 3916 (time 1787709677.05 total 60.07 last 57.94) - open/close 4488 (time 1787709687.07 total 70.08 last 57.10) - open/close 5080 (time 1787709697.08 total 80.10 last 59.11) - open/close 5810 (time 1787709707.09 total 90.11 last 72.90) - open/close 6441 (time 1787709717.10 total 100.12 last 63.04) - open/close 7010 (time 1787709727.11 total 110.13 last 56.84) - open/close 7590 (time 1787709737.13 total 120.14 last 57.93) - open/close 8303 (time 1787709747.13 total 130.15 last 71.24) - open/close 9078 (time 1787709757.13 total 140.15 last 77.50) - open/close 9770 (time 1787709767.14 total 150.16 last 69.14) total: 10000 open/close in 153.66 seconds: 65.08 ops/second - unlinked 0 (time 1787709776 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 617 (time 1787709871.57 total 10.01 last 61.62) - open/close 1237 (time 1787709881.58 total 20.03 last 61.92) - open/close 1919 (time 1787709891.60 total 30.04 last 68.11) - open/close 2595 (time 1787709901.60 total 40.04 last 67.56) - open/close 3378 (time 1787709911.61 total 50.05 last 78.22) - open/close 4076 (time 1787709921.62 total 60.06 last 69.77) - open/close 4742 (time 1787709931.62 total 70.06 last 66.59) - open/close 5436 (time 1787709941.62 total 80.06 last 69.37) - open/close 6140 (time 1787709951.63 total 90.07 last 70.35) - open/close 6874 (time 1787709961.64 total 100.08 last 73.32) - open/close 7699 (time 1787709971.64 total 110.09 last 82.47) - open/close 8364 (time 1787709981.65 total 120.09 last 66.46) - open/close 9133 (time 1787709991.65 total 130.10 last 76.86) - open/close 9908 (time 1787710001.67 total 140.12 last 77.35) total: 10000 open/close in 141.89 seconds: 70.48 ops/second - unlinked 0 (time 1787710008 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 895 (time 1787710103.10 total 10.00 last 89.48) - open/close 1722 (time 1787710113.11 total 20.01 last 82.62) - open/close 2605 (time 1787710123.12 total 30.02 last 88.22) - open/close 3408 (time 1787710133.13 total 40.02 last 80.29) - open/close 4171 (time 1787710143.18 total 50.07 last 75.92) - open/close 4919 (time 1787710153.19 total 60.09 last 74.70) - open/close 5707 (time 1787710163.19 total 70.09 last 78.76) - open/close 6484 (time 1787710173.20 total 80.09 last 77.70) - open/close 7180 (time 1787710183.20 total 90.10 last 69.56) - open/close 7987 (time 1787710193.21 total 100.10 last 80.66) - open/close 8781 (time 1787710203.22 total 110.11 last 79.31) - open/close 9569 (time 1787710213.24 total 120.14 last 78.61) total: 10000 open/close in 126.89 seconds: 78.81 ops/second - unlinked 0 (time 1787710226 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 562 (time 1787710342.50 total 10.00 last 56.20) - open/close 1101 (time 1787710352.51 total 20.01 last 53.84) - open/close 1799 (time 1787710362.52 total 30.02 last 69.76) - open/close 2483 (time 1787710372.52 total 40.02 last 68.39) - open/close 3271 (time 1787710382.52 total 50.03 last 78.75) - open/close 3891 (time 1787710392.54 total 60.04 last 61.94) - open/close 4471 (time 1787710402.56 total 70.06 last 57.88) - open/close 5119 (time 1787710412.56 total 80.06 last 64.80) - open/close 5802 (time 1787710422.57 total 90.07 last 68.24) - open/close 6520 (time 1787710432.58 total 100.08 last 71.71) - open/close 7342 (time 1787710442.58 total 110.08 last 82.18) - open/close 8144 (time 1787710452.59 total 120.09 last 80.11) - open/close 8777 (time 1787710462.59 total 130.09 last 63.30) - open/close 9505 (time 1787710472.60 total 140.10 last 72.73) total: 10000 open/close in 147.34 seconds: 67.87 ops/second - unlinked 0 (time 1787710485 ; total 1 ; last 1) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded. pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2