== conf-sanity test 69: replace an OST with the same index ========================================================== 18:27:04 (1763335624) start mds service on oleg120-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 709 (time 1763335720.25 total 10.02 last 70.72) - open/close 1444 (time 1763335730.25 total 20.03 last 73.47) - open/close 2125 (time 1763335740.25 total 30.03 last 68.08) - open/close 2851 (time 1763335750.26 total 40.04 last 72.56) - open/close 3604 (time 1763335760.26 total 50.04 last 75.27) - open/close 4318 (time 1763335770.27 total 60.04 last 71.38) - open/close 5044 (time 1763335780.27 total 70.05 last 72.55) - open/close 5795 (time 1763335790.28 total 80.05 last 75.07) - open/close 6505 (time 1763335800.29 total 90.07 last 70.91) - open/close 7248 (time 1763335810.30 total 100.08 last 74.23) - open/close 8036 (time 1763335820.30 total 110.08 last 78.79) - open/close 8803 (time 1763335830.31 total 120.09 last 76.64) - open/close 9563 (time 1763335840.31 total 130.09 last 75.95) total: 10000 open/close in 136.51 seconds: 73.26 ops/second - unlinked 0 (time 1763335851 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 764 (time 1763335939.82 total 10.00 last 76.38) - open/close 1531 (time 1763335949.83 total 20.01 last 76.67) - open/close 2278 (time 1763335959.84 total 30.02 last 74.61) - open/close 3005 (time 1763335969.87 total 40.05 last 72.51) - open/close 3757 (time 1763335979.87 total 50.05 last 75.17) - open/close 4534 (time 1763335989.87 total 60.05 last 77.69) - open/close 5329 (time 1763335999.88 total 70.06 last 79.46) - open/close 6089 (time 1763336009.88 total 80.06 last 75.99) - open/close 6850 (time 1763336019.88 total 90.06 last 76.09) - open/close 7693 (time 1763336029.88 total 100.06 last 84.29) - open/close 8551 (time 1763336039.89 total 110.07 last 85.72) - open/close 9376 (time 1763336049.90 total 120.08 last 82.38) total: 10000 open/close in 127.89 seconds: 78.19 ops/second - unlinked 0 (time 1763336062 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 775 (time 1763336151.88 total 10.02 last 77.34) - open/close 1526 (time 1763336161.90 total 20.04 last 74.98) - open/close 2317 (time 1763336171.91 total 30.04 last 79.06) - open/close 3116 (time 1763336181.91 total 40.05 last 79.87) - open/close 3881 (time 1763336191.92 total 50.06 last 76.41) - open/close 4624 (time 1763336201.93 total 60.07 last 74.19) - open/close 5393 (time 1763336211.94 total 70.08 last 76.85) - open/close 6241 (time 1763336221.94 total 80.08 last 84.79) - open/close 7054 (time 1763336231.96 total 90.09 last 81.18) - open/close 8001 (time 1763336241.96 total 100.10 last 94.65) - open/close 8824 (time 1763336251.97 total 110.10 last 82.27) - open/close 9635 (time 1763336261.97 total 120.11 last 81.06) total: 10000 open/close in 124.94 seconds: 80.04 ops/second - unlinked 0 (time 1763336270 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 728 (time 1763336359.34 total 10.00 last 72.77) - open/close 1497 (time 1763336369.34 total 20.01 last 76.86) - open/close 2271 (time 1763336379.35 total 30.02 last 77.35) - open/close 3003 (time 1763336389.36 total 40.03 last 73.10) - open/close 3692 (time 1763336399.36 total 50.03 last 68.90) - open/close 4507 (time 1763336409.37 total 60.04 last 81.43) - open/close 5311 (time 1763336419.38 total 70.04 last 80.36) - open/close 6087 (time 1763336429.38 total 80.05 last 77.56) - open/close 6859 (time 1763336439.39 total 90.06 last 77.13) - open/close 7610 (time 1763336449.41 total 100.08 last 74.98) - open/close 8409 (time 1763336459.41 total 110.08 last 79.86) - open/close 9222 (time 1763336469.43 total 120.09 last 81.19) total: 10000 open/close in 129.83 seconds: 77.02 ops/second - unlinked 0 (time 1763336482 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 792 (time 1763336566.82 total 10.00 last 79.18) - open/close 1601 (time 1763336576.83 total 20.01 last 80.85) - open/close 2400 (time 1763336586.83 total 30.01 last 79.86) - open/close 3183 (time 1763336596.84 total 40.02 last 78.27) - open/close 3966 (time 1763336606.84 total 50.02 last 78.27) - open/close 4717 (time 1763336616.84 total 60.02 last 75.09) - open/close 5529 (time 1763336626.85 total 70.03 last 81.15) - open/close 6339 (time 1763336636.86 total 80.04 last 80.93) - open/close 7134 (time 1763336646.86 total 90.04 last 79.45) - open/close 8004 (time 1763336656.87 total 100.05 last 86.95) - open/close 8822 (time 1763336666.88 total 110.06 last 81.73) - open/close 9652 (time 1763336676.88 total 120.06 last 83.00) total: 10000 open/close in 124.69 seconds: 80.20 ops/second - unlinked 0 (time 1763336685 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 778 (time 1763336765.82 total 10.01 last 77.70) - open/close 1565 (time 1763336775.83 total 20.02 last 78.64) - open/close 2346 (time 1763336785.83 total 30.02 last 78.09) - open/close 3132 (time 1763336795.84 total 40.03 last 78.54) - open/close 3895 (time 1763336805.85 total 50.04 last 76.21) - open/close 4717 (time 1763336815.85 total 60.05 last 82.15) - open/close 5457 (time 1763336825.86 total 70.06 last 73.93) - open/close 6213 (time 1763336835.88 total 80.07 last 75.51) - open/close 7041 (time 1763336845.88 total 90.07 last 82.78) - open/close 8011 (time 1763336855.88 total 100.08 last 96.95) - open/close 8935 (time 1763336865.89 total 110.08 last 92.36) - open/close 9877 (time 1763336875.90 total 120.09 last 94.13) total: 10000 open/close in 121.53 seconds: 82.28 ops/second - unlinked 0 (time 1763336880 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 799 (time 1763336964.05 total 10.01 last 79.83) - open/close 1589 (time 1763336974.05 total 20.01 last 78.99) - open/close 2378 (time 1763336984.06 total 30.03 last 78.77) - open/close 3138 (time 1763336994.07 total 40.03 last 75.95) - open/close 3915 (time 1763337004.09 total 50.05 last 77.54) - open/close 4645 (time 1763337014.09 total 60.06 last 72.98) - open/close 5420 (time 1763337024.10 total 70.06 last 77.45) - open/close 6165 (time 1763337034.11 total 80.08 last 74.41) - open/close 7001 (time 1763337044.12 total 90.08 last 83.53) - open/close 7766 (time 1763337054.12 total 100.09 last 76.47) - open/close 8436 (time 1763337064.13 total 110.09 last 66.96) - open/close 9183 (time 1763337074.13 total 120.10 last 74.67) - open/close 9984 (time 1763337084.14 total 130.10 last 80.09) total: 10000 open/close in 130.27 seconds: 76.76 ops/second - unlinked 0 (time 1763337088 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 739 (time 1763337181.19 total 10.01 last 73.83) - open/close 1483 (time 1763337191.19 total 20.01 last 74.39) - open/close 2259 (time 1763337201.19 total 30.01 last 77.59) - open/close 2988 (time 1763337211.20 total 40.02 last 72.84) - open/close 3767 (time 1763337221.21 total 50.03 last 77.86) - open/close 4568 (time 1763337231.22 total 60.04 last 80.02) - open/close 5385 (time 1763337241.22 total 70.04 last 81.64) - open/close 6195 (time 1763337251.22 total 80.04 last 80.99) - open/close 7005 (time 1763337261.24 total 90.06 last 80.88) - open/close 7837 (time 1763337271.24 total 100.06 last 83.16) - open/close 8623 (time 1763337281.25 total 110.07 last 78.55) - open/close 9421 (time 1763337291.25 total 120.07 last 79.80) total: 10000 open/close in 126.56 seconds: 79.01 ops/second - unlinked 0 (time 1763337300 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 773 (time 1763337390.95 total 10.01 last 77.23) - open/close 1583 (time 1763337400.97 total 20.03 last 80.85) - open/close 2359 (time 1763337410.98 total 30.04 last 77.54) - open/close 3129 (time 1763337420.99 total 40.04 last 76.94) - open/close 3890 (time 1763337430.99 total 50.05 last 76.06) - open/close 4616 (time 1763337440.99 total 60.05 last 72.58) - open/close 5404 (time 1763337451.00 total 70.06 last 78.72) - open/close 6177 (time 1763337461.01 total 80.07 last 77.23) - open/close 6973 (time 1763337471.03 total 90.08 last 79.50) - open/close 7762 (time 1763337481.03 total 100.09 last 78.85) - open/close 8531 (time 1763337491.05 total 110.11 last 76.79) - open/close 9312 (time 1763337501.05 total 120.11 last 78.06) - open/close 9975 (time 1763337511.07 total 130.12 last 66.20) total: 10000 open/close in 130.62 seconds: 76.56 ops/second - unlinked 0 (time 1763337516 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 794 (time 1763337603.28 total 10.01 last 79.35) - open/close 1526 (time 1763337613.29 total 20.01 last 73.15) - open/close 2307 (time 1763337623.30 total 30.02 last 78.03) - open/close 3170 (time 1763337633.31 total 40.03 last 86.22) - open/close 3979 (time 1763337643.31 total 50.03 last 80.89) - open/close 4797 (time 1763337653.31 total 60.03 last 81.78) - open/close 5505 (time 1763337663.31 total 70.04 last 70.77) - open/close 6264 (time 1763337673.33 total 80.06 last 75.75) - open/close 7057 (time 1763337683.35 total 90.07 last 79.18) - open/close 7811 (time 1763337693.36 total 100.09 last 75.29) - open/close 8444 (time 1763337703.37 total 110.09 last 63.28) - open/close 9166 (time 1763337713.38 total 120.11 last 72.09) - open/close 9809 (time 1763337723.40 total 130.13 last 64.16) total: 10000 open/close in 132.97 seconds: 75.20 ops/second - unlinked 0 (time 1763337729 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg120-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2