== conf-sanity test 69: replace an OST with the same index ========================================================== 13:46:49 (1780336009) start mds service on oleg315-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 848 (time 1780336106.36 total 10.00 last 84.80) - open/close 1684 (time 1780336116.36 total 20.01 last 83.53) - open/close 2599 (time 1780336126.36 total 30.01 last 91.49) - open/close 3445 (time 1780336136.37 total 40.01 last 84.59) - open/close 4401 (time 1780336146.37 total 50.02 last 95.53) - open/close 5314 (time 1780336156.39 total 60.03 last 91.18) - open/close 6195 (time 1780336166.40 total 70.04 last 88.02) - open/close 7099 (time 1780336176.40 total 80.05 last 90.33) - open/close 7941 (time 1780336186.40 total 90.05 last 84.20) - open/close 8858 (time 1780336196.41 total 100.05 last 91.67) - open/close 9713 (time 1780336206.42 total 110.06 last 85.39) total: 10000 open/close in 113.34 seconds: 88.23 ops/second - unlinked 0 (time 1780336213 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 919 (time 1780336288.92 total 10.00 last 91.87) - open/close 1804 (time 1780336298.92 total 20.01 last 88.45) - open/close 2810 (time 1780336308.93 total 30.01 last 100.54) - open/close 3722 (time 1780336318.93 total 40.02 last 91.17) - open/close 4596 (time 1780336328.94 total 50.03 last 87.33) - open/close 5513 (time 1780336338.94 total 60.03 last 91.65) - open/close 6455 (time 1780336348.95 total 70.04 last 94.13) - open/close 7407 (time 1780336358.96 total 80.04 last 95.16) - open/close 8307 (time 1780336368.96 total 90.05 last 89.92) - open/close 9258 (time 1780336378.97 total 100.06 last 95.06) total: 10000 open/close in 108.46 seconds: 92.20 ops/second - unlinked 0 (time 1780336390 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 807 (time 1780336476.32 total 10.01 last 80.64) - open/close 1677 (time 1780336486.32 total 20.01 last 87.00) - open/close 2616 (time 1780336496.34 total 30.03 last 93.71) - open/close 3503 (time 1780336506.36 total 40.05 last 88.51) - open/close 4389 (time 1780336516.37 total 50.06 last 88.53) - open/close 5272 (time 1780336526.37 total 60.06 last 88.27) - open/close 6131 (time 1780336536.38 total 70.07 last 85.85) - open/close 6928 (time 1780336546.39 total 80.08 last 79.62) - open/close 7678 (time 1780336556.39 total 90.08 last 74.99) - open/close 8341 (time 1780336566.39 total 100.08 last 66.29) - open/close 9031 (time 1780336576.40 total 110.09 last 68.94) - open/close 9781 (time 1780336586.41 total 120.10 last 74.90) total: 10000 open/close in 122.34 seconds: 81.74 ops/second - unlinked 0 (time 1780336593 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 890 (time 1780336672.20 total 10.01 last 88.90) - open/close 1783 (time 1780336682.21 total 20.02 last 89.20) - open/close 2648 (time 1780336692.22 total 30.03 last 86.46) - open/close 3648 (time 1780336702.22 total 40.03 last 99.93) - open/close 4612 (time 1780336712.22 total 50.04 last 96.39) - open/close 5538 (time 1780336722.23 total 60.04 last 92.59) - open/close 6496 (time 1780336732.24 total 70.05 last 95.68) - open/close 7400 (time 1780336742.24 total 80.05 last 90.37) - open/close 8238 (time 1780336752.26 total 90.07 last 83.61) - open/close 9191 (time 1780336762.27 total 100.08 last 95.20) total: 10000 open/close in 109.66 seconds: 91.19 ops/second - unlinked 0 (time 1780336776 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 882 (time 1780336865.60 total 10.01 last 88.13) - open/close 1733 (time 1780336875.60 total 20.01 last 85.09) - open/close 2566 (time 1780336885.60 total 30.01 last 83.25) - open/close 3291 (time 1780336895.62 total 40.03 last 72.40) - open/close 4308 (time 1780336905.63 total 50.04 last 101.55) - open/close 5325 (time 1780336915.63 total 60.05 last 101.67) - open/close 6258 (time 1780336925.64 total 70.05 last 93.27) - open/close 7200 (time 1780336935.64 total 80.05 last 94.19) - open/close 8157 (time 1780336945.64 total 90.05 last 95.66) - open/close 8963 (time 1780336955.65 total 100.06 last 80.56) - open/close 9747 (time 1780336965.65 total 110.06 last 78.36) total: 10000 open/close in 112.82 seconds: 88.64 ops/second - unlinked 0 (time 1780336972 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 998 (time 1780337050.91 total 10.01 last 99.74) - open/close 1876 (time 1780337060.92 total 20.01 last 87.75) - open/close 2880 (time 1780337070.93 total 30.03 last 100.26) - open/close 3818 (time 1780337080.94 total 40.03 last 93.73) - open/close 4796 (time 1780337090.94 total 50.04 last 97.77) - open/close 5837 (time 1780337100.95 total 60.04 last 104.02) - open/close 6644 (time 1780337110.95 total 70.05 last 80.68) - open/close 7584 (time 1780337120.96 total 80.06 last 93.91) - open/close 8399 (time 1780337130.96 total 90.06 last 81.48) - open/close 9118 (time 1780337140.97 total 100.07 last 71.83) total: 10000 open/close in 109.70 seconds: 91.16 ops/second - unlinked 0 (time 1780337154 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 625 (time 1780337234.80 total 10.01 last 62.44) - open/close 1429 (time 1780337244.82 total 20.03 last 80.22) - open/close 2031 (time 1780337254.83 total 30.04 last 60.17) - open/close 2708 (time 1780337264.83 total 40.04 last 67.68) - open/close 3684 (time 1780337274.84 total 50.04 last 97.55) - open/close 4512 (time 1780337284.84 total 60.05 last 82.80) - open/close 5343 (time 1780337294.84 total 70.05 last 83.07) - open/close 6177 (time 1780337304.85 total 80.06 last 83.34) - open/close 7059 (time 1780337314.86 total 90.06 last 88.13) - open/close 8066 (time 1780337324.86 total 100.07 last 100.68) - open/close 8986 (time 1780337334.87 total 110.08 last 91.91) - open/close 9858 (time 1780337344.88 total 120.09 last 87.06) total: 10000 open/close in 121.64 seconds: 82.21 ops/second - unlinked 0 (time 1780337350 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 791 (time 1780337435.63 total 10.00 last 79.07) - open/close 1694 (time 1780337445.63 total 20.01 last 90.27) - open/close 2582 (time 1780337455.63 total 30.01 last 88.79) - open/close 3464 (time 1780337465.64 total 40.01 last 88.15) - open/close 4371 (time 1780337475.64 total 50.02 last 90.68) - open/close 5101 (time 1780337485.65 total 60.02 last 72.95) - open/close 5941 (time 1780337495.66 total 70.04 last 83.88) - open/close 6807 (time 1780337505.66 total 80.04 last 86.56) - open/close 7516 (time 1780337515.68 total 90.05 last 70.82) - open/close 8312 (time 1780337525.68 total 100.06 last 79.56) - open/close 9171 (time 1780337535.69 total 110.07 last 85.82) total: 10000 open/close in 119.97 seconds: 83.35 ops/second - unlinked 0 (time 1780337549 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 889 (time 1780337633.82 total 10.01 last 88.78) - open/close 1709 (time 1780337643.82 total 20.02 last 81.99) - open/close 2266 (time 1780337653.85 total 30.04 last 55.57) - open/close 2745 (time 1780337663.86 total 40.05 last 47.83) - open/close 3267 (time 1780337673.88 total 50.07 last 52.12) - open/close 4064 (time 1780337683.89 total 60.08 last 79.63) - open/close 4904 (time 1780337693.90 total 70.09 last 83.91) - open/close 5757 (time 1780337703.92 total 80.11 last 85.11) - open/close 6607 (time 1780337713.93 total 90.12 last 84.90) - open/close 7335 (time 1780337723.94 total 100.13 last 72.75) - open/close 7979 (time 1780337733.94 total 110.14 last 64.35) - open/close 8844 (time 1780337743.95 total 120.15 last 86.42) - open/close 9695 (time 1780337753.96 total 130.15 last 85.08) total: 10000 open/close in 133.48 seconds: 74.92 ops/second - unlinked 0 (time 1780337761 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 764 (time 1780337850.15 total 10.01 last 76.34) - open/close 1471 (time 1780337860.15 total 20.01 last 70.67) - open/close 2184 (time 1780337870.16 total 30.02 last 71.25) - open/close 2899 (time 1780337880.17 total 40.04 last 71.38) - open/close 3652 (time 1780337890.18 total 50.05 last 75.23) - open/close 4286 (time 1780337900.19 total 60.05 last 63.35) - open/close 5079 (time 1780337910.20 total 70.06 last 79.25) - open/close 5870 (time 1780337920.20 total 80.07 last 79.05) - open/close 6700 (time 1780337930.22 total 90.08 last 82.85) - open/close 7548 (time 1780337940.24 total 100.10 last 84.68) - open/close 8357 (time 1780337950.24 total 110.10 last 80.89) - open/close 9133 (time 1780337960.24 total 120.10 last 77.57) - open/close 9979 (time 1780337970.25 total 130.11 last 84.54) total: 10000 open/close in 130.31 seconds: 76.74 ops/second - unlinked 0 (time 1780337974 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.115@tcp sys.timeout=20 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded. pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2