== conf-sanity test 69: replace an OST with the same index ========================================================== 01:14:35 (1769753675) start mds service on oleg354-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg354-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg354-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 795 (time 1769753786.12 total 10.01 last 79.40) - open/close 1455 (time 1769753796.13 total 20.02 last 65.95) - open/close 2474 (time 1769753806.14 total 30.03 last 101.80) - open/close 3184 (time 1769753816.16 total 40.05 last 70.88) - open/close 3956 (time 1769753826.17 total 50.06 last 77.11) - open/close 4828 (time 1769753836.17 total 60.06 last 87.17) - open/close 5780 (time 1769753846.18 total 70.07 last 95.16) - open/close 6568 (time 1769753856.18 total 80.07 last 78.79) - open/close 7167 (time 1769753866.19 total 90.08 last 59.82) - open/close 7909 (time 1769753876.19 total 100.08 last 74.19) - open/close 8695 (time 1769753886.20 total 110.09 last 78.55) - open/close 9546 (time 1769753896.20 total 120.10 last 85.04) total: 10000 open/close in 127.28 seconds: 78.57 ops/second - unlinked 0 (time 1769753908 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 880 (time 1769753997.11 total 10.01 last 87.93) - open/close 1770 (time 1769754007.11 total 20.01 last 88.97) - open/close 2532 (time 1769754017.12 total 30.01 last 76.17) - open/close 3122 (time 1769754027.13 total 40.02 last 58.95) - open/close 3908 (time 1769754037.14 total 50.04 last 78.48) - open/close 4534 (time 1769754047.16 total 60.05 last 62.52) - open/close 5276 (time 1769754057.16 total 70.06 last 74.14) - open/close 6021 (time 1769754067.17 total 80.06 last 74.47) - open/close 6775 (time 1769754077.19 total 90.08 last 75.25) - open/close 7497 (time 1769754087.19 total 100.08 last 72.20) - open/close 8350 (time 1769754097.20 total 110.09 last 85.23) - open/close 9155 (time 1769754107.21 total 120.10 last 80.40) - open/close 9977 (time 1769754117.22 total 130.12 last 82.10) total: 10000 open/close in 130.52 seconds: 76.62 ops/second - unlinked 0 (time 1769754122 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 550 (time 1769754218.43 total 10.03 last 54.86) - open/close 1125 (time 1769754228.44 total 20.04 last 57.40) - open/close 1709 (time 1769754238.45 total 30.05 last 58.39) - open/close 2545 (time 1769754248.46 total 40.06 last 83.48) - open/close 3225 (time 1769754258.47 total 50.07 last 67.91) - open/close 3909 (time 1769754268.48 total 60.08 last 68.34) - open/close 4715 (time 1769754278.49 total 70.09 last 80.55) - open/close 5434 (time 1769754288.50 total 80.10 last 71.86) - open/close 6348 (time 1769754298.50 total 90.10 last 91.37) - open/close 7148 (time 1769754308.51 total 100.11 last 79.93) - open/close 8013 (time 1769754318.52 total 110.12 last 86.36) - open/close 8761 (time 1769754328.53 total 120.13 last 74.71) - open/close 9600 (time 1769754338.53 total 130.13 last 83.90) total: 10000 open/close in 134.07 seconds: 74.59 ops/second - unlinked 0 (time 1769754346 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1035 (time 1769754435.21 total 10.00 last 103.48) - open/close 2002 (time 1769754445.21 total 20.01 last 96.65) - open/close 2930 (time 1769754455.22 total 30.01 last 92.73) - open/close 3894 (time 1769754465.23 total 40.02 last 96.33) - open/close 4868 (time 1769754475.23 total 50.03 last 97.36) - open/close 5823 (time 1769754485.24 total 60.03 last 95.47) - open/close 6691 (time 1769754495.24 total 70.04 last 86.73) - open/close 7411 (time 1769754505.25 total 80.05 last 71.93) - open/close 8210 (time 1769754515.26 total 90.05 last 79.88) - open/close 9099 (time 1769754525.26 total 100.05 last 88.85) - open/close 9854 (time 1769754535.26 total 110.06 last 75.48) total: 10000 open/close in 111.89 seconds: 89.38 ops/second - unlinked 0 (time 1769754541 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 954 (time 1769754634.58 total 10.01 last 95.34) - open/close 1763 (time 1769754644.58 total 20.01 last 80.86) - open/close 2455 (time 1769754654.59 total 30.01 last 69.19) - open/close 3284 (time 1769754664.62 total 40.05 last 82.62) - open/close 4188 (time 1769754674.62 total 50.05 last 90.38) - open/close 5012 (time 1769754684.63 total 60.05 last 82.37) - open/close 5828 (time 1769754694.64 total 70.06 last 81.52) - open/close 6673 (time 1769754704.64 total 80.07 last 84.43) - open/close 7516 (time 1769754714.66 total 90.09 last 84.12) - open/close 8371 (time 1769754724.67 total 100.09 last 85.48) - open/close 9230 (time 1769754734.67 total 110.10 last 85.89) total: 10000 open/close in 117.86 seconds: 84.84 ops/second - unlinked 0 (time 1769754747 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 517 (time 1769754834.08 total 10.03 last 51.56) - open/close 1086 (time 1769754844.08 total 20.03 last 56.87) - open/close 1978 (time 1769754854.08 total 30.03 last 89.20) - open/close 2876 (time 1769754864.09 total 40.04 last 89.75) - open/close 3761 (time 1769754874.09 total 50.04 last 88.50) - open/close 4477 (time 1769754884.09 total 60.04 last 71.59) - open/close 5279 (time 1769754894.10 total 70.05 last 80.10) - open/close 6020 (time 1769754904.12 total 80.06 last 74.01) - open/close 6771 (time 1769754914.12 total 90.07 last 75.05) - open/close 7647 (time 1769754924.13 total 100.08 last 87.51) - open/close 8553 (time 1769754934.13 total 110.08 last 90.60) - open/close 9391 (time 1769754944.14 total 120.08 last 83.77) - open/close 9994 (time 1769754954.14 total 130.09 last 60.27) total: 10000 open/close in 130.17 seconds: 76.82 ops/second - unlinked 0 (time 1769754960 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 928 (time 1769755049.33 total 10.01 last 92.73) - open/close 1792 (time 1769755059.33 total 20.01 last 86.35) - open/close 2697 (time 1769755069.34 total 30.02 last 90.42) - open/close 3490 (time 1769755079.36 total 40.04 last 79.16) - open/close 4394 (time 1769755089.37 total 50.05 last 90.35) - open/close 5211 (time 1769755099.37 total 60.05 last 81.67) - open/close 6054 (time 1769755109.38 total 70.06 last 84.25) - open/close 6839 (time 1769755119.38 total 80.06 last 78.44) - open/close 7529 (time 1769755129.39 total 90.07 last 68.95) - open/close 8360 (time 1769755139.40 total 100.08 last 83.06) - open/close 9211 (time 1769755149.40 total 110.08 last 85.07) total: 10000 open/close in 117.69 seconds: 84.97 ops/second - unlinked 0 (time 1769755161 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 798 (time 1769755246.29 total 10.00 last 79.78) - open/close 1792 (time 1769755256.30 total 20.00 last 99.38) - open/close 2884 (time 1769755266.30 total 30.01 last 109.14) - open/close 3968 (time 1769755276.31 total 40.01 last 108.35) - open/close 4863 (time 1769755286.33 total 50.03 last 89.32) - open/close 5905 (time 1769755296.33 total 60.04 last 104.18) - open/close 6890 (time 1769755306.34 total 70.05 last 98.41) - open/close 7756 (time 1769755316.34 total 80.05 last 86.58) - open/close 8642 (time 1769755326.35 total 90.05 last 88.55) - open/close 9508 (time 1769755336.36 total 100.06 last 86.51) total: 10000 open/close in 105.87 seconds: 94.46 ops/second - unlinked 0 (time 1769755346 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 898 (time 1769755425.84 total 10.00 last 89.79) - open/close 1826 (time 1769755435.85 total 20.01 last 92.71) - open/close 2763 (time 1769755445.85 total 30.01 last 93.67) - open/close 3742 (time 1769755455.87 total 40.03 last 97.79) - open/close 4591 (time 1769755465.88 total 50.04 last 84.82) - open/close 5427 (time 1769755475.88 total 60.04 last 83.56) - open/close 6232 (time 1769755485.88 total 70.04 last 80.47) - open/close 7115 (time 1769755495.89 total 80.05 last 88.28) - open/close 7912 (time 1769755505.89 total 90.05 last 79.64) - open/close 8661 (time 1769755515.90 total 100.06 last 74.85) - open/close 9566 (time 1769755525.90 total 110.06 last 90.49) total: 10000 open/close in 114.86 seconds: 87.06 ops/second - unlinked 0 (time 1769755535 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 887 (time 1769755610.63 total 10.02 last 88.54) - open/close 1654 (time 1769755620.65 total 20.04 last 76.54) - open/close 2464 (time 1769755630.66 total 30.04 last 80.96) - open/close 3348 (time 1769755640.66 total 40.05 last 88.34) - open/close 4256 (time 1769755650.67 total 50.06 last 90.70) - open/close 5280 (time 1769755660.69 total 60.07 last 102.28) - open/close 6088 (time 1769755670.70 total 70.08 last 80.73) - open/close 6973 (time 1769755680.70 total 80.09 last 88.49) - open/close 7731 (time 1769755690.70 total 90.09 last 75.79) - open/close 8675 (time 1769755700.71 total 100.10 last 94.26) - open/close 9636 (time 1769755710.72 total 110.11 last 96.03) total: 10000 open/close in 113.52 seconds: 88.09 ops/second - unlinked 0 (time 1769755718 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg354-server: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg354-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg354-server: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg354-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.154@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.154@tcp sys.timeout=20 start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg354-server: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg354-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg354-server: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg354-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg354-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2