== conf-sanity test 69: replace an OST with the same index ========================================================== 10:26:55 (1786976815) start mds service on oleg158-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg158-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 714 (time 1786976942.39 total 10.00 last 71.37) - open/close 1414 (time 1786976952.40 total 20.01 last 69.96) - open/close 2114 (time 1786976962.41 total 30.02 last 69.94) - open/close 2853 (time 1786976972.41 total 40.02 last 73.90) - open/close 3521 (time 1786976982.44 total 50.05 last 66.57) - open/close 4211 (time 1786976992.46 total 60.07 last 68.91) - open/close 4864 (time 1786977002.48 total 70.09 last 65.17) - open/close 5578 (time 1786977012.49 total 80.10 last 71.32) - open/close 6302 (time 1786977022.49 total 90.10 last 72.36) - open/close 6892 (time 1786977032.52 total 100.13 last 58.86) - open/close 7668 (time 1786977042.52 total 110.13 last 77.56) - open/close 8466 (time 1786977052.53 total 120.14 last 79.75) - open/close 9104 (time 1786977062.53 total 130.14 last 63.75) - open/close 9992 (time 1786977072.56 total 140.17 last 88.57) total: 10000 open/close in 140.33 seconds: 71.26 ops/second - unlinked 0 (time 1786977077 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 756 (time 1786977168.41 total 10.01 last 75.56) - open/close 1481 (time 1786977178.42 total 20.01 last 72.43) - open/close 2292 (time 1786977188.42 total 30.01 last 81.10) - open/close 3134 (time 1786977198.44 total 40.03 last 84.07) - open/close 3765 (time 1786977208.44 total 50.04 last 63.06) - open/close 4463 (time 1786977218.45 total 60.04 last 69.75) - open/close 5310 (time 1786977228.45 total 70.05 last 84.68) - open/close 6056 (time 1786977238.46 total 80.05 last 74.53) - open/close 6737 (time 1786977248.47 total 90.07 last 68.01) - open/close 7280 (time 1786977258.48 total 100.08 last 54.26) - open/close 7852 (time 1786977268.48 total 110.08 last 57.18) - open/close 8465 (time 1786977278.49 total 120.09 last 61.26) - open/close 9111 (time 1786977288.49 total 130.09 last 64.59) - open/close 9797 (time 1786977298.50 total 140.10 last 68.54) total: 10000 open/close in 143.18 seconds: 69.84 ops/second - unlinked 0 (time 1786977306 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 803 (time 1786977398.92 total 10.00 last 80.27) - open/close 1628 (time 1786977408.92 total 20.01 last 82.46) - open/close 2386 (time 1786977418.93 total 30.01 last 75.78) - open/close 3279 (time 1786977428.94 total 40.02 last 89.19) - open/close 3968 (time 1786977438.96 total 50.05 last 68.73) - open/close 4722 (time 1786977448.97 total 60.05 last 75.36) - open/close 5359 (time 1786977458.97 total 70.05 last 63.70) - open/close 6140 (time 1786977468.98 total 80.06 last 78.03) - open/close 6943 (time 1786977479.00 total 90.08 last 80.13) - open/close 7707 (time 1786977489.00 total 100.09 last 76.38) - open/close 8523 (time 1786977499.00 total 110.09 last 81.60) - open/close 9427 (time 1786977509.01 total 120.10 last 90.31) total: 10000 open/close in 129.26 seconds: 77.36 ops/second - unlinked 0 (time 1786977523 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 633 (time 1786977619.09 total 10.00 last 63.30) - open/close 1267 (time 1786977629.09 total 20.01 last 63.36) - open/close 2237 (time 1786977639.10 total 30.01 last 96.96) - open/close 3088 (time 1786977649.10 total 40.02 last 85.03) - open/close 4061 (time 1786977659.12 total 50.03 last 97.16) - open/close 4698 (time 1786977669.13 total 60.04 last 63.63) - open/close 5372 (time 1786977679.14 total 70.05 last 67.35) - open/close 5990 (time 1786977689.14 total 80.05 last 61.78) - open/close 6701 (time 1786977699.14 total 90.06 last 71.06) - open/close 7601 (time 1786977709.15 total 100.07 last 89.91) - open/close 8497 (time 1786977719.16 total 110.07 last 89.56) - open/close 9206 (time 1786977729.17 total 120.09 last 70.81) total: 10000 open/close in 129.24 seconds: 77.38 ops/second - unlinked 0 (time 1786977744 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 691 (time 1786977824.80 total 10.00 last 69.09) - open/close 1482 (time 1786977834.81 total 20.02 last 78.98) - open/close 2372 (time 1786977844.82 total 30.02 last 88.96) - open/close 3170 (time 1786977854.82 total 40.02 last 79.78) - open/close 3910 (time 1786977864.83 total 50.03 last 73.95) - open/close 4825 (time 1786977874.83 total 60.04 last 91.45) - open/close 5706 (time 1786977884.84 total 70.04 last 88.04) - open/close 6730 (time 1786977894.85 total 80.05 last 102.31) - open/close 7544 (time 1786977904.86 total 90.06 last 81.34) - open/close 8483 (time 1786977914.86 total 100.06 last 93.84) - open/close 9526 (time 1786977924.87 total 110.07 last 104.25) total: 10000 open/close in 114.30 seconds: 87.49 ops/second - unlinked 0 (time 1786977933 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 979 (time 1786978016.35 total 10.01 last 97.81) - open/close 1802 (time 1786978026.36 total 20.02 last 82.20) - open/close 2691 (time 1786978036.37 total 30.03 last 88.82) - open/close 3505 (time 1786978046.37 total 40.04 last 81.36) - open/close 4465 (time 1786978056.38 total 50.04 last 95.96) - open/close 5432 (time 1786978066.38 total 60.04 last 96.66) - open/close 6428 (time 1786978076.38 total 70.05 last 99.57) - open/close 7225 (time 1786978086.39 total 80.06 last 79.62) - open/close 8030 (time 1786978096.40 total 90.06 last 80.44) - open/close 8838 (time 1786978106.42 total 100.08 last 80.67) - open/close 9456 (time 1786978116.42 total 110.09 last 61.77) total: 10000 open/close in 117.90 seconds: 84.82 ops/second - unlinked 0 (time 1786978128 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1142 (time 1786978208.09 total 10.00 last 114.16) - open/close 2030 (time 1786978218.10 total 20.01 last 88.73) - open/close 3020 (time 1786978228.10 total 30.01 last 99.00) - open/close 4098 (time 1786978238.10 total 40.02 last 107.74) - open/close 5077 (time 1786978248.11 total 50.02 last 97.84) - open/close 5981 (time 1786978258.11 total 60.03 last 90.36) - open/close 6886 (time 1786978268.12 total 70.03 last 90.44) - open/close 7871 (time 1786978278.13 total 80.05 last 98.40) - open/close 8980 (time 1786978288.13 total 90.05 last 110.88) total: 10000 open/close in 99.10 seconds: 100.91 ops/second - unlinked 0 (time 1786978301 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 931 (time 1786978372.94 total 10.01 last 92.99) - open/close 1649 (time 1786978382.95 total 20.02 last 71.76) - open/close 2485 (time 1786978392.99 total 30.06 last 83.24) - open/close 3396 (time 1786978403.00 total 40.07 last 91.03) - open/close 4486 (time 1786978413.00 total 50.07 last 108.94) - open/close 5192 (time 1786978423.01 total 60.08 last 70.59) - open/close 6004 (time 1786978433.01 total 70.08 last 81.18) - open/close 7006 (time 1786978443.01 total 80.08 last 100.14) - open/close 7812 (time 1786978453.02 total 90.09 last 80.51) - open/close 8559 (time 1786978463.03 total 100.10 last 74.67) - open/close 9571 (time 1786978473.03 total 110.10 last 101.16) total: 10000 open/close in 115.07 seconds: 86.90 ops/second - unlinked 0 (time 1786978482 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1115 (time 1786978559.07 total 10.00 last 111.48) - open/close 2043 (time 1786978569.07 total 20.00 last 92.79) - open/close 2995 (time 1786978579.08 total 30.01 last 95.15) - open/close 3964 (time 1786978589.09 total 40.02 last 96.79) - open/close 4853 (time 1786978599.09 total 50.02 last 88.89) - open/close 5835 (time 1786978609.09 total 60.02 last 98.17) - open/close 6875 (time 1786978619.10 total 70.04 last 103.86) - open/close 7811 (time 1786978629.11 total 80.04 last 93.53) - open/close 8762 (time 1786978639.13 total 90.06 last 94.98) - open/close 9819 (time 1786978649.13 total 100.06 last 105.64) total: 10000 open/close in 103.55 seconds: 96.57 ops/second - unlinked 0 (time 1786978658 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 950 (time 1786978732.12 total 10.01 last 94.94) - open/close 1817 (time 1786978742.12 total 20.01 last 86.67) - open/close 2627 (time 1786978752.13 total 30.02 last 80.95) - open/close 3439 (time 1786978762.13 total 40.02 last 81.19) - open/close 4001 (time 1786978772.13 total 50.02 last 56.16) - open/close 4865 (time 1786978782.14 total 60.03 last 86.33) - open/close 5671 (time 1786978792.15 total 70.04 last 80.55) - open/close 6590 (time 1786978802.15 total 80.04 last 91.88) - open/close 7644 (time 1786978812.16 total 90.05 last 105.33) - open/close 8742 (time 1786978822.16 total 100.05 last 109.73) - open/close 9801 (time 1786978832.16 total 110.05 last 105.90) total: 10000 open/close in 112.91 seconds: 88.57 ops/second - unlinked 0 (time 1786978839 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg158-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg158-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.158@tcp sys.timeout=20 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg158-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg158-server: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg158-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: modules unloaded. pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2