== conf-sanity test 69: replace an OST with the same index ========================================================== 03:19:15 (1780384755) start mds service on oleg153-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg153-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg153-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 759 (time 1780384860.57 total 10.01 last 75.83) - open/close 1473 (time 1780384870.57 total 20.01 last 71.37) - open/close 2331 (time 1780384880.58 total 30.02 last 85.71) - open/close 3089 (time 1780384890.59 total 40.03 last 75.76) - open/close 3878 (time 1780384900.59 total 50.03 last 78.87) - open/close 4701 (time 1780384910.62 total 60.06 last 82.11) - open/close 5445 (time 1780384920.63 total 70.07 last 74.33) - open/close 6264 (time 1780384930.64 total 80.08 last 81.76) - open/close 7063 (time 1780384940.65 total 90.09 last 79.85) - open/close 7982 (time 1780384950.66 total 100.10 last 91.86) - open/close 8762 (time 1780384960.66 total 110.10 last 77.95) - open/close 9478 (time 1780384970.68 total 120.12 last 71.50) total: 10000 open/close in 128.30 seconds: 77.94 ops/second - unlinked 0 (time 1780384984 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 874 (time 1780385076.27 total 10.01 last 87.27) - open/close 1784 (time 1780385086.27 total 20.02 last 91.00) - open/close 2676 (time 1780385096.27 total 30.02 last 89.14) - open/close 3486 (time 1780385106.27 total 40.02 last 80.99) - open/close 4287 (time 1780385116.28 total 50.03 last 80.04) - open/close 5052 (time 1780385126.29 total 60.04 last 76.42) - open/close 5999 (time 1780385136.30 total 70.05 last 94.64) - open/close 7045 (time 1780385146.30 total 80.05 last 104.57) - open/close 7905 (time 1780385156.32 total 90.07 last 85.86) - open/close 8852 (time 1780385166.33 total 100.08 last 94.61) - open/close 9738 (time 1780385176.35 total 110.10 last 88.42) total: 10000 open/close in 113.45 seconds: 88.14 ops/second - unlinked 0 (time 1780385183 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 747 (time 1780385270.40 total 10.01 last 74.65) - open/close 1619 (time 1780385280.40 total 20.01 last 87.20) - open/close 2376 (time 1780385290.41 total 30.01 last 75.64) - open/close 3241 (time 1780385300.42 total 40.02 last 86.45) - open/close 4153 (time 1780385310.42 total 50.02 last 91.19) - open/close 4993 (time 1780385320.43 total 60.03 last 83.93) - open/close 5835 (time 1780385330.44 total 70.04 last 84.09) - open/close 6636 (time 1780385340.44 total 80.04 last 80.10) - open/close 7494 (time 1780385350.44 total 90.05 last 85.77) - open/close 7852 (time 1780385360.47 total 100.07 last 35.71) - open/close 8360 (time 1780385370.47 total 110.08 last 50.78) - open/close 8958 (time 1780385380.49 total 120.09 last 59.72) - open/close 9504 (time 1780385390.49 total 130.10 last 54.56) total: 10000 open/close in 137.41 seconds: 72.77 ops/second - unlinked 0 (time 1780385404 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 698 (time 1780385499.67 total 10.01 last 69.76) - open/close 1388 (time 1780385509.69 total 20.03 last 68.86) - open/close 2092 (time 1780385519.69 total 30.03 last 70.38) - open/close 2843 (time 1780385529.70 total 40.03 last 75.07) - open/close 3726 (time 1780385539.71 total 50.04 last 88.22) - open/close 4472 (time 1780385549.71 total 60.04 last 74.59) - open/close 5284 (time 1780385559.71 total 70.05 last 81.18) - open/close 6111 (time 1780385569.71 total 80.05 last 82.70) - open/close 7001 (time 1780385579.72 total 90.05 last 88.93) - open/close 7824 (time 1780385589.72 total 100.06 last 82.25) - open/close 8670 (time 1780385599.73 total 110.07 last 84.52) - open/close 9386 (time 1780385609.74 total 120.07 last 71.58) total: 10000 open/close in 128.23 seconds: 77.99 ops/second - unlinked 0 (time 1780385622 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 481 (time 1780385707.75 total 10.01 last 48.05) - open/close 1210 (time 1780385717.75 total 20.01 last 72.90) - open/close 2213 (time 1780385727.76 total 30.02 last 100.24) - open/close 3011 (time 1780385737.76 total 40.02 last 79.77) - open/close 3853 (time 1780385747.77 total 50.02 last 84.16) - open/close 4736 (time 1780385757.78 total 60.04 last 88.18) - open/close 5632 (time 1780385767.78 total 70.04 last 89.59) - open/close 6505 (time 1780385777.78 total 80.04 last 87.29) - open/close 7393 (time 1780385787.79 total 90.05 last 88.72) - open/close 8281 (time 1780385797.80 total 100.06 last 88.70) - open/close 9214 (time 1780385807.80 total 110.06 last 93.27) total: 10000 open/close in 118.36 seconds: 84.48 ops/second - unlinked 0 (time 1780385819 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 887 (time 1780385894.50 total 10.00 last 88.68) - open/close 1862 (time 1780385904.50 total 20.01 last 97.45) - open/close 2824 (time 1780385914.51 total 30.01 last 96.16) - open/close 3902 (time 1780385924.52 total 40.02 last 107.68) - open/close 4949 (time 1780385934.52 total 50.03 last 104.66) - open/close 5984 (time 1780385944.52 total 60.03 last 103.50) - open/close 6946 (time 1780385954.53 total 70.03 last 96.13) - open/close 7965 (time 1780385964.53 total 80.03 last 101.89) - open/close 8951 (time 1780385974.54 total 90.04 last 98.53) total: 10000 open/close in 100.02 seconds: 99.98 ops/second - unlinked 0 (time 1780385987 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 965 (time 1780386059.40 total 10.01 last 96.43) - open/close 1934 (time 1780386069.40 total 20.01 last 96.85) - open/close 2821 (time 1780386079.42 total 30.03 last 88.54) - open/close 3773 (time 1780386089.43 total 40.04 last 95.08) - open/close 4785 (time 1780386099.44 total 50.05 last 101.08) - open/close 5739 (time 1780386109.44 total 60.05 last 95.40) - open/close 6673 (time 1780386119.47 total 70.08 last 93.20) - open/close 7660 (time 1780386129.48 total 80.09 last 98.54) - open/close 8624 (time 1780386139.49 total 90.10 last 96.33) - open/close 9639 (time 1780386149.50 total 100.11 last 101.35) total: 10000 open/close in 104.10 seconds: 96.06 ops/second - unlinked 0 (time 1780386156 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 939 (time 1780386226.69 total 10.01 last 93.85) - open/close 1947 (time 1780386236.70 total 20.01 last 100.76) - open/close 3041 (time 1780386246.71 total 30.02 last 109.28) - open/close 4101 (time 1780386256.71 total 40.02 last 105.96) - open/close 5074 (time 1780386266.71 total 50.02 last 97.29) - open/close 5941 (time 1780386276.72 total 60.03 last 86.64) - open/close 6875 (time 1780386286.73 total 70.04 last 93.33) - open/close 7924 (time 1780386296.73 total 80.05 last 104.84) - open/close 8919 (time 1780386306.74 total 90.05 last 99.44) - open/close 9936 (time 1780386316.74 total 100.05 last 101.67) total: 10000 open/close in 100.68 seconds: 99.33 ops/second - unlinked 0 (time 1780386320 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1041 (time 1780386387.33 total 10.00 last 104.08) - open/close 2153 (time 1780386397.34 total 20.01 last 111.06) - open/close 3221 (time 1780386407.35 total 30.02 last 106.73) - open/close 4288 (time 1780386417.36 total 40.03 last 106.60) - open/close 5369 (time 1780386427.36 total 50.03 last 108.10) - open/close 6592 (time 1780386437.37 total 60.04 last 122.20) - open/close 7797 (time 1780386447.37 total 70.04 last 120.44) - open/close 8856 (time 1780386457.38 total 80.05 last 105.86) - open/close 9939 (time 1780386467.38 total 90.05 last 108.23) total: 10000 open/close in 90.66 seconds: 110.31 ops/second - unlinked 0 (time 1780386470 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1150 (time 1780386532.94 total 10.01 last 114.94) - open/close 2428 (time 1780386542.96 total 20.02 last 127.67) - open/close 3581 (time 1780386552.96 total 30.02 last 115.24) - open/close 4746 (time 1780386562.97 total 40.03 last 116.40) - open/close 5910 (time 1780386572.98 total 50.04 last 116.28) - open/close 7121 (time 1780386582.98 total 60.04 last 121.05) - open/close 8333 (time 1780386592.99 total 70.05 last 121.17) - open/close 9459 (time 1780386602.99 total 80.05 last 112.59) total: 10000 open/close in 84.70 seconds: 118.06 ops/second - unlinked 0 (time 1780386610 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre On OST0, 16303 used inodes rc=0 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.153@tcp sys.timeout=20 start ost1 service on oleg153-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg153-server: oleg153-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg153-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local oleg153-server: modules unloaded. pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2