== conf-sanity test 69: replace an OST with the same index ========================================================== 01:48:38 (1761284918) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 780 (time 1761285009.40 total 10.01 last 77.91) - open/close 1536 (time 1761285019.40 total 20.01 last 75.60) - open/close 2407 (time 1761285029.41 total 30.02 last 87.05) - open/close 3274 (time 1761285039.42 total 40.03 last 86.61) - open/close 4056 (time 1761285049.42 total 50.03 last 78.19) - open/close 4810 (time 1761285059.42 total 60.03 last 75.38) - open/close 5592 (time 1761285069.42 total 70.04 last 78.17) - open/close 6327 (time 1761285079.43 total 80.04 last 73.45) - open/close 7101 (time 1761285089.44 total 90.05 last 77.34) - open/close 7927 (time 1761285099.44 total 100.05 last 82.60) - open/close 8715 (time 1761285109.44 total 110.05 last 78.78) - open/close 9440 (time 1761285119.44 total 120.05 last 72.50) total: 10000 open/close in 127.23 seconds: 78.60 ops/second - unlinked 0 (time 1761285130 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 831 (time 1761285210.67 total 10.00 last 83.09) - open/close 1642 (time 1761285220.68 total 20.01 last 81.02) - open/close 2384 (time 1761285230.69 total 30.02 last 74.13) - open/close 3154 (time 1761285240.70 total 40.03 last 76.91) - open/close 3939 (time 1761285250.70 total 50.03 last 78.50) - open/close 4811 (time 1761285260.71 total 60.04 last 87.14) - open/close 5677 (time 1761285270.71 total 70.04 last 86.60) - open/close 6502 (time 1761285280.72 total 80.05 last 82.42) - open/close 7356 (time 1761285290.73 total 90.06 last 85.32) - open/close 8118 (time 1761285300.73 total 100.06 last 76.19) - open/close 8952 (time 1761285310.74 total 110.06 last 83.35) - open/close 9706 (time 1761285320.74 total 120.07 last 75.38) total: 10000 open/close in 124.12 seconds: 80.57 ops/second - unlinked 0 (time 1761285328 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 780 (time 1761285418.82 total 10.01 last 77.92) - open/close 1518 (time 1761285428.83 total 20.02 last 73.76) - open/close 2255 (time 1761285438.83 total 30.02 last 73.68) - open/close 2992 (time 1761285448.83 total 40.02 last 73.69) - open/close 3765 (time 1761285458.84 total 50.02 last 77.28) - open/close 4526 (time 1761285468.85 total 60.03 last 76.04) - open/close 5295 (time 1761285478.86 total 70.05 last 76.79) - open/close 6076 (time 1761285488.87 total 80.05 last 78.05) - open/close 6920 (time 1761285498.87 total 90.06 last 84.32) - open/close 7808 (time 1761285508.88 total 100.06 last 88.77) - open/close 8788 (time 1761285518.88 total 110.07 last 97.97) - open/close 9763 (time 1761285528.89 total 120.08 last 97.42) total: 10000 open/close in 123.02 seconds: 81.29 ops/second - unlinked 0 (time 1761285534 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 860 (time 1761285611.39 total 10.00 last 85.97) - open/close 1777 (time 1761285621.40 total 20.01 last 91.60) - open/close 2636 (time 1761285631.41 total 30.03 last 85.80) - open/close 3469 (time 1761285641.42 total 40.03 last 83.26) - open/close 4330 (time 1761285651.43 total 50.04 last 86.03) - open/close 5167 (time 1761285661.45 total 60.06 last 83.52) - open/close 5938 (time 1761285671.46 total 70.08 last 76.98) - open/close 6726 (time 1761285681.47 total 80.08 last 78.76) - open/close 7518 (time 1761285691.48 total 90.09 last 79.14) - open/close 8300 (time 1761285701.49 total 100.10 last 78.13) - open/close 9069 (time 1761285711.49 total 110.10 last 76.87) - open/close 9824 (time 1761285721.50 total 120.11 last 75.41) total: 10000 open/close in 122.42 seconds: 81.68 ops/second - unlinked 0 (time 1761285727 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 746 (time 1761285814.67 total 10.03 last 74.41) - open/close 1534 (time 1761285824.68 total 20.03 last 78.74) - open/close 2376 (time 1761285834.68 total 30.04 last 84.19) - open/close 3161 (time 1761285844.68 total 40.04 last 78.45) - open/close 3954 (time 1761285854.70 total 50.05 last 79.21) - open/close 4675 (time 1761285864.70 total 60.06 last 72.08) - open/close 5387 (time 1761285874.70 total 70.06 last 71.18) - open/close 6321 (time 1761285884.70 total 80.06 last 93.38) - open/close 7221 (time 1761285894.71 total 90.07 last 89.94) - open/close 8055 (time 1761285904.71 total 100.07 last 83.38) - open/close 8861 (time 1761285914.72 total 110.07 last 80.58) - open/close 9631 (time 1761285924.72 total 120.07 last 76.98) total: 10000 open/close in 125.01 seconds: 79.99 ops/second - unlinked 0 (time 1761285933 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 759 (time 1761286015.09 total 10.01 last 75.79) - open/close 1571 (time 1761286025.10 total 20.02 last 81.12) - open/close 2368 (time 1761286035.10 total 30.03 last 79.68) - open/close 3185 (time 1761286045.12 total 40.05 last 81.54) - open/close 3938 (time 1761286055.13 total 50.06 last 75.20) - open/close 4679 (time 1761286065.14 total 60.06 last 74.10) - open/close 5495 (time 1761286075.14 total 70.07 last 81.55) - open/close 6331 (time 1761286085.15 total 80.07 last 83.56) - open/close 7113 (time 1761286095.15 total 90.07 last 78.20) - open/close 7943 (time 1761286105.15 total 100.07 last 82.98) - open/close 8770 (time 1761286115.15 total 110.07 last 82.70) - open/close 9655 (time 1761286125.16 total 120.08 last 88.42) total: 10000 open/close in 124.01 seconds: 80.64 ops/second - unlinked 0 (time 1761286132 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 817 (time 1761286215.17 total 10.01 last 81.64) - open/close 1677 (time 1761286225.18 total 20.01 last 85.99) - open/close 2496 (time 1761286235.18 total 30.01 last 81.89) - open/close 3339 (time 1761286245.18 total 40.01 last 84.28) - open/close 4145 (time 1761286255.19 total 50.02 last 80.55) - open/close 4947 (time 1761286265.19 total 60.03 last 80.15) - open/close 5700 (time 1761286275.20 total 70.03 last 75.24) - open/close 6448 (time 1761286285.21 total 80.04 last 74.75) - open/close 7210 (time 1761286295.21 total 90.05 last 76.16) - open/close 7988 (time 1761286305.22 total 100.06 last 77.72) - open/close 8714 (time 1761286315.24 total 110.08 last 72.44) - open/close 9417 (time 1761286325.26 total 120.10 last 70.17) total: 10000 open/close in 128.25 seconds: 77.97 ops/second - unlinked 0 (time 1761286337 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 665 (time 1761286431.31 total 10.00 last 66.48) - open/close 1317 (time 1761286441.31 total 20.00 last 65.19) - open/close 2071 (time 1761286451.32 total 30.01 last 75.35) - open/close 2837 (time 1761286461.32 total 40.01 last 76.59) - open/close 3566 (time 1761286471.33 total 50.02 last 72.82) - open/close 4337 (time 1761286481.34 total 60.04 last 77.02) - open/close 5116 (time 1761286491.34 total 70.04 last 77.87) - open/close 5833 (time 1761286501.35 total 80.04 last 71.68) - open/close 6543 (time 1761286511.36 total 90.06 last 70.89) - open/close 7253 (time 1761286521.37 total 100.06 last 70.98) - open/close 7993 (time 1761286531.39 total 110.08 last 73.86) - open/close 8718 (time 1761286541.40 total 120.09 last 72.42) - open/close 9452 (time 1761286551.40 total 130.10 last 73.36) total: 10000 open/close in 137.26 seconds: 72.85 ops/second - unlinked 0 (time 1761286561 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 720 (time 1761286648.82 total 10.02 last 71.82) - open/close 1390 (time 1761286658.83 total 20.03 last 66.95) - open/close 2118 (time 1761286668.83 total 30.03 last 72.79) - open/close 2940 (time 1761286678.84 total 40.05 last 82.10) - open/close 3708 (time 1761286688.84 total 50.05 last 76.75) - open/close 4433 (time 1761286698.86 total 60.07 last 72.40) - open/close 5116 (time 1761286708.87 total 70.08 last 68.22) - open/close 5795 (time 1761286718.87 total 80.08 last 67.90) - open/close 6471 (time 1761286728.89 total 90.10 last 67.46) - open/close 7201 (time 1761286738.90 total 100.11 last 72.95) - open/close 7885 (time 1761286748.90 total 110.11 last 68.37) - open/close 8649 (time 1761286758.91 total 120.11 last 76.38) - open/close 9366 (time 1761286768.92 total 130.12 last 71.63) total: 10000 open/close in 138.81 seconds: 72.04 ops/second - unlinked 0 (time 1761286779 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 661 (time 1761286869.24 total 10.01 last 66.03) - open/close 1318 (time 1761286879.25 total 20.01 last 65.68) - open/close 1935 (time 1761286889.26 total 30.02 last 61.64) - open/close 2538 (time 1761286899.26 total 40.03 last 60.25) - open/close 3178 (time 1761286909.27 total 50.04 last 63.96) - open/close 3846 (time 1761286919.28 total 60.05 last 66.72) - open/close 4544 (time 1761286929.29 total 70.06 last 69.73) - open/close 5243 (time 1761286939.31 total 80.07 last 69.82) - open/close 5926 (time 1761286949.31 total 90.08 last 68.26) - open/close 6648 (time 1761286959.31 total 100.08 last 72.18) - open/close 7328 (time 1761286969.32 total 110.09 last 67.92) - open/close 7998 (time 1761286979.33 total 120.10 last 66.94) - open/close 8697 (time 1761286989.34 total 130.10 last 69.89) - open/close 9417 (time 1761286999.34 total 140.10 last 71.98) total: 10000 open/close in 148.29 seconds: 67.44 ops/second - unlinked 0 (time 1761287010 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.118@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2