== conf-sanity test 69: replace an OST with the same index ========================================================== 20:15:18 (1770686118) start mds service on oleg123-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg123-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg123-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 763 (time 1770686229.47 total 10.01 last 76.21) - open/close 1228 (time 1770686239.48 total 20.02 last 46.44) - open/close 1855 (time 1770686249.49 total 30.03 last 62.64) - open/close 2571 (time 1770686259.50 total 40.04 last 71.55) - open/close 3274 (time 1770686269.51 total 50.05 last 70.26) - open/close 4027 (time 1770686279.52 total 60.06 last 75.22) - open/close 4678 (time 1770686289.52 total 70.06 last 65.07) - open/close 5370 (time 1770686299.53 total 80.06 last 69.17) - open/close 6106 (time 1770686309.54 total 90.08 last 73.50) - open/close 6841 (time 1770686319.55 total 100.09 last 73.44) - open/close 7564 (time 1770686329.56 total 110.10 last 72.21) - open/close 8301 (time 1770686339.57 total 120.11 last 73.65) - open/close 9033 (time 1770686349.59 total 130.13 last 73.06) - open/close 9742 (time 1770686359.59 total 140.13 last 70.87) total: 10000 open/close in 144.44 seconds: 69.23 ops/second - unlinked 0 (time 1770686369 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 682 (time 1770686466.35 total 10.00 last 68.17) - open/close 1411 (time 1770686476.35 total 20.01 last 72.89) - open/close 2126 (time 1770686486.37 total 30.02 last 71.36) - open/close 2842 (time 1770686496.38 total 40.03 last 71.58) - open/close 3558 (time 1770686506.39 total 50.04 last 71.53) - open/close 4318 (time 1770686516.39 total 60.04 last 75.99) - open/close 4869 (time 1770686526.39 total 70.04 last 55.07) - open/close 5653 (time 1770686536.39 total 80.05 last 78.38) - open/close 6357 (time 1770686546.39 total 90.05 last 70.40) - open/close 7075 (time 1770686556.41 total 100.06 last 71.72) - open/close 7759 (time 1770686566.41 total 110.06 last 68.36) - open/close 8488 (time 1770686576.41 total 120.06 last 72.90) - open/close 9179 (time 1770686586.42 total 130.07 last 69.03) - open/close 9798 (time 1770686596.44 total 140.09 last 61.79) total: 10000 open/close in 143.46 seconds: 69.71 ops/second - unlinked 0 (time 1770686604 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 753 (time 1770686701.38 total 10.01 last 75.23) - open/close 1385 (time 1770686711.41 total 20.03 last 63.04) - open/close 2101 (time 1770686721.42 total 30.05 last 71.52) - open/close 2817 (time 1770686731.43 total 40.05 last 71.54) - open/close 3491 (time 1770686741.43 total 50.06 last 67.36) - open/close 4148 (time 1770686751.44 total 60.07 last 65.66) - open/close 4905 (time 1770686761.45 total 70.07 last 75.65) - open/close 5551 (time 1770686771.45 total 80.08 last 64.55) - open/close 6255 (time 1770686781.46 total 90.08 last 70.36) - open/close 6958 (time 1770686791.46 total 100.09 last 70.28) - open/close 7695 (time 1770686801.47 total 110.09 last 73.66) - open/close 8423 (time 1770686811.47 total 120.10 last 72.75) - open/close 9212 (time 1770686821.49 total 130.11 last 78.81) - open/close 9883 (time 1770686831.49 total 140.12 last 67.06) total: 10000 open/close in 141.48 seconds: 70.68 ops/second - unlinked 0 (time 1770686837 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 768 (time 1770686932.07 total 10.01 last 76.71) - open/close 1471 (time 1770686942.07 total 20.02 last 70.26) - open/close 2271 (time 1770686952.07 total 30.02 last 79.99) - open/close 2968 (time 1770686962.08 total 40.02 last 69.67) - open/close 3785 (time 1770686972.08 total 50.03 last 81.68) - open/close 4578 (time 1770686982.08 total 60.03 last 79.29) - open/close 5363 (time 1770686992.09 total 70.04 last 78.43) - open/close 6139 (time 1770687002.10 total 80.05 last 77.50) - open/close 6920 (time 1770687012.11 total 90.05 last 78.06) - open/close 7732 (time 1770687022.11 total 100.06 last 81.16) - open/close 8576 (time 1770687032.12 total 110.06 last 84.38) - open/close 9386 (time 1770687042.12 total 120.06 last 81.00) total: 10000 open/close in 127.81 seconds: 78.24 ops/second - unlinked 0 (time 1770687053 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 760 (time 1770687139.77 total 10.01 last 75.92) - open/close 1692 (time 1770687149.77 total 20.02 last 93.13) - open/close 2549 (time 1770687159.77 total 30.02 last 85.68) - open/close 3374 (time 1770687169.79 total 40.03 last 82.41) - open/close 4234 (time 1770687179.79 total 50.03 last 85.97) - open/close 5107 (time 1770687189.79 total 60.04 last 87.26) - open/close 5892 (time 1770687199.81 total 70.05 last 78.39) - open/close 6802 (time 1770687209.82 total 80.06 last 90.92) - open/close 7564 (time 1770687219.82 total 90.07 last 76.17) - open/close 8467 (time 1770687229.82 total 100.07 last 90.27) - open/close 9380 (time 1770687239.83 total 110.07 last 91.27) total: 10000 open/close in 117.30 seconds: 85.25 ops/second - unlinked 0 (time 1770687250 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 870 (time 1770687329.81 total 10.00 last 86.98) - open/close 1734 (time 1770687339.81 total 20.00 last 86.39) - open/close 2672 (time 1770687349.82 total 30.00 last 93.79) - open/close 3523 (time 1770687359.82 total 40.01 last 85.06) - open/close 4378 (time 1770687369.83 total 50.02 last 85.42) - open/close 5262 (time 1770687379.84 total 60.02 last 88.34) - open/close 6107 (time 1770687389.84 total 70.03 last 84.50) - open/close 6952 (time 1770687399.85 total 80.04 last 84.38) - open/close 7809 (time 1770687409.86 total 90.05 last 85.65) - open/close 8591 (time 1770687419.87 total 100.05 last 78.13) - open/close 9501 (time 1770687429.87 total 110.06 last 90.98) total: 10000 open/close in 115.03 seconds: 86.93 ops/second - unlinked 0 (time 1770687437 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 922 (time 1770687516.87 total 10.00 last 92.16) - open/close 1734 (time 1770687526.88 total 20.01 last 81.15) - open/close 2575 (time 1770687536.89 total 30.02 last 84.04) - open/close 3474 (time 1770687546.89 total 40.02 last 89.86) - open/close 4126 (time 1770687556.90 total 50.03 last 65.13) - open/close 5028 (time 1770687566.91 total 60.04 last 90.14) - open/close 5878 (time 1770687576.91 total 70.05 last 84.94) - open/close 6730 (time 1770687586.92 total 80.05 last 85.19) - open/close 7603 (time 1770687596.92 total 90.05 last 87.29) - open/close 8457 (time 1770687606.92 total 100.06 last 85.33) - open/close 9340 (time 1770687616.93 total 110.06 last 88.28) total: 10000 open/close in 117.27 seconds: 85.27 ops/second - unlinked 0 (time 1770687626 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 860 (time 1770687699.55 total 10.00 last 85.99) - open/close 1708 (time 1770687709.55 total 20.01 last 84.74) - open/close 2585 (time 1770687719.56 total 30.01 last 87.67) - open/close 3346 (time 1770687729.58 total 40.04 last 75.93) - open/close 4139 (time 1770687739.58 total 50.04 last 79.29) - open/close 5000 (time 1770687749.59 total 60.05 last 86.02) - open/close 5878 (time 1770687759.60 total 70.05 last 87.72) - open/close 6777 (time 1770687769.60 total 80.05 last 89.90) - open/close 7666 (time 1770687779.60 total 90.06 last 88.85) - open/close 8538 (time 1770687789.61 total 100.06 last 87.16) - open/close 9456 (time 1770687799.61 total 110.07 last 91.75) total: 10000 open/close in 116.34 seconds: 85.95 ops/second - unlinked 0 (time 1770687809 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 924 (time 1770687886.33 total 10.00 last 92.39) - open/close 1843 (time 1770687896.34 total 20.00 last 91.87) - open/close 2754 (time 1770687906.36 total 30.02 last 90.93) - open/close 3636 (time 1770687916.36 total 40.03 last 88.13) - open/close 4502 (time 1770687926.37 total 50.03 last 86.57) - open/close 5412 (time 1770687936.37 total 60.04 last 90.97) - open/close 6483 (time 1770687946.37 total 70.04 last 107.09) - open/close 7407 (time 1770687956.37 total 80.04 last 92.39) - open/close 8320 (time 1770687966.38 total 90.05 last 91.24) - open/close 9142 (time 1770687976.38 total 100.05 last 82.16) total: 10000 open/close in 108.64 seconds: 92.05 ops/second - unlinked 0 (time 1770687987 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 984 (time 1770688053.69 total 10.01 last 98.31) - open/close 1959 (time 1770688063.69 total 20.01 last 97.48) - open/close 2796 (time 1770688073.70 total 30.02 last 83.64) - open/close 3621 (time 1770688083.71 total 40.02 last 82.47) - open/close 4539 (time 1770688093.71 total 50.02 last 91.77) - open/close 5444 (time 1770688103.71 total 60.03 last 90.48) - open/close 6361 (time 1770688113.71 total 70.03 last 91.67) - open/close 7305 (time 1770688123.72 total 80.04 last 94.35) - open/close 8120 (time 1770688133.72 total 90.04 last 81.47) - open/close 9019 (time 1770688143.72 total 100.04 last 89.89) - open/close 9910 (time 1770688153.73 total 110.05 last 89.04) total: 10000 open/close in 110.93 seconds: 90.15 ops/second - unlinked 0 (time 1770688157 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre On OST0, 13641 used inodes rc=0 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.123@tcp sys.timeout=20 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg123-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2