== conf-sanity test 69: replace an OST with the same index ========================================================== 18:51:59 (1763337119) start mds service on oleg107-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg107-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 752 (time 1763337213.30 total 10.01 last 75.13) - open/close 1473 (time 1763337223.30 total 20.01 last 72.09) - open/close 2259 (time 1763337233.30 total 30.01 last 78.57) - open/close 3079 (time 1763337243.31 total 40.03 last 81.89) - open/close 3910 (time 1763337253.32 total 50.03 last 83.07) - open/close 4734 (time 1763337263.33 total 60.04 last 82.33) - open/close 5578 (time 1763337273.34 total 70.05 last 84.31) - open/close 6344 (time 1763337283.34 total 80.06 last 76.57) - open/close 7197 (time 1763337293.35 total 90.06 last 85.24) - open/close 8064 (time 1763337303.35 total 100.07 last 86.66) - open/close 8860 (time 1763337313.36 total 110.07 last 79.57) - open/close 9617 (time 1763337323.36 total 120.07 last 75.70) total: 10000 open/close in 125.30 seconds: 79.81 ops/second - unlinked 0 (time 1763337332 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 703 (time 1763337422.54 total 10.03 last 70.12) - open/close 1442 (time 1763337432.55 total 20.03 last 73.86) - open/close 2195 (time 1763337442.55 total 30.04 last 75.27) - open/close 2955 (time 1763337452.57 total 40.05 last 75.90) - open/close 3709 (time 1763337462.57 total 50.06 last 75.34) - open/close 4456 (time 1763337472.58 total 60.06 last 74.66) - open/close 5273 (time 1763337482.58 total 70.06 last 81.68) - open/close 6079 (time 1763337492.59 total 80.07 last 80.57) - open/close 6855 (time 1763337502.59 total 90.07 last 77.55) - open/close 7575 (time 1763337512.61 total 100.09 last 71.90) - open/close 8285 (time 1763337522.61 total 110.09 last 70.96) - open/close 9010 (time 1763337532.62 total 120.10 last 72.45) - open/close 9830 (time 1763337542.62 total 130.11 last 81.96) total: 10000 open/close in 132.20 seconds: 75.64 ops/second - unlinked 0 (time 1763337548 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 854 (time 1763337634.77 total 10.00 last 85.36) - open/close 1647 (time 1763337644.77 total 20.01 last 79.27) - open/close 2447 (time 1763337654.78 total 30.02 last 79.95) - open/close 3183 (time 1763337664.79 total 40.02 last 73.54) - open/close 3947 (time 1763337674.80 total 50.03 last 76.32) - open/close 4761 (time 1763337684.80 total 60.04 last 81.36) - open/close 5528 (time 1763337694.81 total 70.05 last 76.63) - open/close 6167 (time 1763337704.83 total 80.07 last 63.80) - open/close 6859 (time 1763337714.84 total 90.07 last 69.16) - open/close 7631 (time 1763337724.84 total 100.07 last 77.18) - open/close 8403 (time 1763337734.85 total 110.09 last 77.10) - open/close 9170 (time 1763337744.85 total 120.09 last 76.69) - open/close 9897 (time 1763337754.85 total 130.09 last 72.68) total: 10000 open/close in 131.30 seconds: 76.16 ops/second - unlinked 0 (time 1763337759 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 772 (time 1763337847.93 total 10.01 last 77.12) - open/close 1561 (time 1763337857.94 total 20.01 last 78.88) - open/close 2354 (time 1763337867.94 total 30.02 last 79.28) - open/close 3119 (time 1763337877.96 total 40.03 last 76.38) - open/close 3878 (time 1763337887.96 total 50.04 last 75.87) - open/close 4694 (time 1763337897.96 total 60.04 last 81.59) - open/close 5460 (time 1763337907.96 total 70.04 last 76.59) - open/close 6240 (time 1763337917.98 total 80.06 last 77.87) - open/close 7040 (time 1763337927.99 total 90.06 last 79.95) - open/close 7837 (time 1763337938.00 total 100.07 last 79.63) - open/close 8636 (time 1763337948.00 total 110.08 last 79.83) - open/close 9420 (time 1763337958.02 total 120.10 last 78.28) total: 10000 open/close in 127.47 seconds: 78.45 ops/second - unlinked 0 (time 1763337969 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 709 (time 1763338060.79 total 10.01 last 70.82) - open/close 1430 (time 1763338070.79 total 20.01 last 72.09) - open/close 2147 (time 1763338080.79 total 30.01 last 71.69) - open/close 2872 (time 1763338090.80 total 40.02 last 72.43) - open/close 3659 (time 1763338100.81 total 50.03 last 78.66) - open/close 4453 (time 1763338110.82 total 60.04 last 79.28) - open/close 5174 (time 1763338120.83 total 70.05 last 72.03) - open/close 5905 (time 1763338130.84 total 80.06 last 73.04) - open/close 6686 (time 1763338140.85 total 90.07 last 78.06) - open/close 7350 (time 1763338150.85 total 100.07 last 66.36) - open/close 8088 (time 1763338160.87 total 110.09 last 73.69) - open/close 8719 (time 1763338170.88 total 120.10 last 63.04) - open/close 9488 (time 1763338180.88 total 130.10 last 76.84) total: 10000 open/close in 136.80 seconds: 73.10 ops/second - unlinked 0 (time 1763338191 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 724 (time 1763338285.36 total 10.00 last 72.38) - open/close 1517 (time 1763338295.36 total 20.00 last 79.30) - open/close 2308 (time 1763338305.37 total 30.01 last 79.02) - open/close 3094 (time 1763338315.37 total 40.02 last 78.59) - open/close 3887 (time 1763338325.38 total 50.02 last 79.26) - open/close 4606 (time 1763338335.38 total 60.02 last 71.87) - open/close 5341 (time 1763338345.39 total 70.03 last 73.48) - open/close 6109 (time 1763338355.39 total 80.03 last 76.77) - open/close 6858 (time 1763338365.39 total 90.03 last 74.90) - open/close 7686 (time 1763338375.39 total 100.03 last 82.77) - open/close 8380 (time 1763338385.40 total 110.04 last 69.37) - open/close 9134 (time 1763338395.40 total 120.04 last 75.38) - open/close 9900 (time 1763338405.41 total 130.05 last 76.55) total: 10000 open/close in 131.43 seconds: 76.09 ops/second - unlinked 0 (time 1763338410 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 791 (time 1763338500.79 total 10.01 last 79.04) - open/close 1515 (time 1763338510.79 total 20.02 last 72.35) - open/close 2241 (time 1763338520.80 total 30.02 last 72.55) - open/close 3014 (time 1763338530.80 total 40.02 last 77.29) - open/close 3781 (time 1763338540.80 total 50.02 last 76.68) - open/close 4595 (time 1763338550.81 total 60.03 last 81.32) - open/close 5370 (time 1763338560.82 total 70.04 last 77.44) - open/close 6108 (time 1763338570.84 total 80.06 last 73.66) - open/close 6815 (time 1763338580.84 total 90.07 last 70.67) - open/close 7537 (time 1763338590.85 total 100.07 last 72.19) - open/close 8302 (time 1763338600.85 total 110.07 last 76.48) - open/close 9055 (time 1763338610.87 total 120.09 last 75.17) - open/close 9812 (time 1763338620.87 total 130.09 last 75.64) total: 10000 open/close in 132.50 seconds: 75.47 ops/second - unlinked 0 (time 1763338626 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 725 (time 1763338717.41 total 10.01 last 72.41) - open/close 1472 (time 1763338727.41 total 20.01 last 74.69) - open/close 2265 (time 1763338737.42 total 30.02 last 79.26) - open/close 3042 (time 1763338747.42 total 40.03 last 77.65) - open/close 3860 (time 1763338757.43 total 50.03 last 81.77) - open/close 4642 (time 1763338767.44 total 60.04 last 78.12) - open/close 5435 (time 1763338777.44 total 70.05 last 79.25) - open/close 6180 (time 1763338787.45 total 80.05 last 74.45) - open/close 6966 (time 1763338797.46 total 90.06 last 78.55) - open/close 7656 (time 1763338807.46 total 100.06 last 68.97) - open/close 8421 (time 1763338817.47 total 110.07 last 76.41) - open/close 9191 (time 1763338827.48 total 120.08 last 76.96) - open/close 9958 (time 1763338837.48 total 130.08 last 76.70) total: 10000 open/close in 130.64 seconds: 76.55 ops/second - unlinked 0 (time 1763338842 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 760 (time 1763338931.34 total 10.00 last 75.99) - open/close 1545 (time 1763338941.35 total 20.01 last 78.43) - open/close 2266 (time 1763338951.37 total 30.02 last 71.99) - open/close 3074 (time 1763338961.37 total 40.03 last 80.76) - open/close 3832 (time 1763338971.37 total 50.03 last 75.79) - open/close 4543 (time 1763338981.38 total 60.04 last 71.08) - open/close 5271 (time 1763338991.38 total 70.04 last 72.79) - open/close 6072 (time 1763339001.38 total 80.04 last 80.09) - open/close 6849 (time 1763339011.39 total 90.04 last 77.65) - open/close 7648 (time 1763339021.39 total 100.05 last 79.85) - open/close 8435 (time 1763339031.39 total 110.05 last 78.68) - open/close 9251 (time 1763339041.40 total 120.05 last 81.59) total: 10000 open/close in 129.70 seconds: 77.10 ops/second - unlinked 0 (time 1763339054 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 753 (time 1763339145.10 total 10.00 last 75.29) - open/close 1538 (time 1763339155.11 total 20.01 last 78.45) - open/close 2364 (time 1763339165.11 total 30.01 last 82.57) - open/close 3188 (time 1763339175.12 total 40.02 last 82.32) - open/close 3953 (time 1763339185.12 total 50.03 last 76.46) - open/close 4759 (time 1763339195.13 total 60.03 last 80.59) - open/close 5510 (time 1763339205.13 total 70.03 last 75.06) - open/close 6328 (time 1763339215.14 total 80.04 last 81.76) - open/close 7163 (time 1763339225.15 total 90.05 last 83.37) - open/close 7920 (time 1763339235.16 total 100.06 last 75.64) - open/close 8756 (time 1763339245.17 total 110.07 last 83.55) - open/close 9500 (time 1763339255.17 total 120.07 last 74.35) total: 10000 open/close in 126.77 seconds: 78.88 ops/second - unlinked 0 (time 1763339265 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg107-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2