== conf-sanity test 69: replace an OST with the same index ========================================================== 20:21:05 (1763342465) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: MODOPTS_LIBCFS= oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 910 (time 1763342544.70 total 10.00 last 91.00) - open/close 1858 (time 1763342554.71 total 20.01 last 94.72) - open/close 2873 (time 1763342564.74 total 30.04 last 101.22) - open/close 3773 (time 1763342574.75 total 40.04 last 89.93) - open/close 4760 (time 1763342584.75 total 50.04 last 98.69) - open/close 5734 (time 1763342594.77 total 60.06 last 97.20) - open/close 6644 (time 1763342604.78 total 70.08 last 90.90) - open/close 7590 (time 1763342614.79 total 80.08 last 94.52) - open/close 8491 (time 1763342624.79 total 90.09 last 90.04) - open/close 9429 (time 1763342634.80 total 100.10 last 93.75) total: 10000 open/close in 106.48 seconds: 93.91 ops/second - unlinked 0 (time 1763342645 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 942 (time 1763342718.29 total 10.01 last 94.14) - open/close 1810 (time 1763342728.30 total 20.01 last 86.74) - open/close 2639 (time 1763342738.31 total 30.02 last 82.84) - open/close 3577 (time 1763342748.31 total 40.02 last 93.77) - open/close 4521 (time 1763342758.31 total 50.02 last 94.39) - open/close 5495 (time 1763342768.31 total 60.03 last 97.36) - open/close 6432 (time 1763342778.32 total 70.03 last 93.66) - open/close 7357 (time 1763342788.33 total 80.04 last 92.42) - open/close 8312 (time 1763342798.33 total 90.05 last 95.44) - open/close 9289 (time 1763342808.33 total 100.05 last 97.69) total: 10000 open/close in 106.89 seconds: 93.55 ops/second - unlinked 0 (time 1763342819 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 899 (time 1763342894.65 total 10.01 last 89.81) - open/close 1851 (time 1763342904.66 total 20.01 last 95.16) - open/close 2877 (time 1763342914.67 total 30.03 last 102.49) - open/close 3911 (time 1763342924.68 total 40.03 last 103.30) - open/close 4879 (time 1763342934.68 total 50.04 last 96.78) - open/close 5881 (time 1763342944.68 total 60.04 last 100.19) - open/close 6856 (time 1763342954.69 total 70.04 last 97.44) - open/close 7854 (time 1763342964.70 total 80.05 last 99.69) - open/close 8808 (time 1763342974.71 total 90.06 last 95.34) - open/close 9731 (time 1763342984.71 total 100.06 last 92.29) total: 10000 open/close in 103.00 seconds: 97.09 ops/second - unlinked 0 (time 1763342991 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 938 (time 1763343065.61 total 10.00 last 93.79) - open/close 1812 (time 1763343075.62 total 20.01 last 87.35) - open/close 2759 (time 1763343085.62 total 30.01 last 94.66) - open/close 3764 (time 1763343095.62 total 40.01 last 100.47) - open/close 4778 (time 1763343105.63 total 50.02 last 101.36) - open/close 5716 (time 1763343115.63 total 60.02 last 93.76) - open/close 6633 (time 1763343125.63 total 70.02 last 91.68) - open/close 7552 (time 1763343135.63 total 80.02 last 91.89) - open/close 8295 (time 1763343145.64 total 90.03 last 74.27) - open/close 9027 (time 1763343155.64 total 100.03 last 73.17) - open/close 9863 (time 1763343165.65 total 110.04 last 83.57) total: 10000 open/close in 111.44 seconds: 89.74 ops/second - unlinked 0 (time 1763343170 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 794 (time 1763343245.27 total 10.01 last 79.30) - open/close 1553 (time 1763343255.29 total 20.04 last 75.70) - open/close 2456 (time 1763343265.30 total 30.04 last 90.26) - open/close 3483 (time 1763343275.31 total 40.05 last 102.63) - open/close 4403 (time 1763343285.31 total 50.06 last 91.94) - open/close 5418 (time 1763343295.31 total 60.06 last 101.50) - open/close 6342 (time 1763343305.32 total 70.07 last 92.33) - open/close 7333 (time 1763343315.33 total 80.07 last 99.03) - open/close 8349 (time 1763343325.33 total 90.08 last 101.55) - open/close 9352 (time 1763343335.35 total 100.09 last 100.13) total: 10000 open/close in 106.35 seconds: 94.03 ops/second - unlinked 0 (time 1763343344 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1046 (time 1763343417.19 total 10.00 last 104.60) - open/close 1991 (time 1763343427.20 total 20.01 last 94.38) - open/close 2888 (time 1763343437.22 total 30.03 last 89.55) - open/close 3888 (time 1763343447.22 total 40.03 last 99.95) - open/close 4872 (time 1763343457.22 total 50.04 last 98.39) - open/close 5800 (time 1763343467.23 total 60.04 last 92.76) - open/close 6827 (time 1763343477.24 total 70.05 last 102.59) - open/close 7813 (time 1763343487.25 total 80.06 last 98.51) - open/close 8789 (time 1763343497.25 total 90.06 last 97.57) - open/close 9774 (time 1763343507.25 total 100.07 last 98.48) total: 10000 open/close in 102.53 seconds: 97.53 ops/second - unlinked 0 (time 1763343513 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 993 (time 1763343585.93 total 10.01 last 99.19) - open/close 1965 (time 1763343595.93 total 20.01 last 97.19) - open/close 2930 (time 1763343605.93 total 30.02 last 96.47) - open/close 3912 (time 1763343615.94 total 40.02 last 98.18) - open/close 4853 (time 1763343625.94 total 50.02 last 94.09) - open/close 5784 (time 1763343635.94 total 60.02 last 93.06) - open/close 6809 (time 1763343645.96 total 70.04 last 102.33) - open/close 7739 (time 1763343655.96 total 80.04 last 92.99) - open/close 8693 (time 1763343665.96 total 90.04 last 95.38) - open/close 9655 (time 1763343675.97 total 100.05 last 96.16) total: 10000 open/close in 103.27 seconds: 96.83 ops/second - unlinked 0 (time 1763343681 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 970 (time 1763343751.88 total 10.01 last 96.88) - open/close 2051 (time 1763343761.89 total 20.02 last 108.03) - open/close 3144 (time 1763343771.89 total 30.02 last 109.24) - open/close 4218 (time 1763343781.89 total 40.03 last 107.37) - open/close 5183 (time 1763343791.90 total 50.03 last 96.48) - open/close 6232 (time 1763343801.90 total 60.03 last 104.85) - open/close 7249 (time 1763343811.91 total 70.04 last 101.63) - open/close 8317 (time 1763343821.91 total 80.05 last 106.75) - open/close 9340 (time 1763343831.92 total 90.05 last 102.26) total: 10000 open/close in 96.24 seconds: 103.91 ops/second - unlinked 0 (time 1763343841 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1030 (time 1763343912.55 total 10.00 last 102.96) - open/close 1919 (time 1763343922.55 total 20.01 last 88.88) - open/close 2980 (time 1763343932.56 total 30.01 last 106.05) - open/close 4053 (time 1763343942.56 total 40.01 last 107.27) - open/close 5103 (time 1763343952.56 total 50.01 last 105.00) - open/close 6102 (time 1763343962.57 total 60.02 last 99.82) - open/close 7144 (time 1763343972.58 total 70.03 last 104.12) - open/close 8126 (time 1763343982.58 total 80.04 last 98.12) - open/close 9119 (time 1763343992.60 total 90.05 last 99.19) total: 10000 open/close in 98.76 seconds: 101.25 ops/second - unlinked 0 (time 1763344005 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 983 (time 1763344078.28 total 10.00 last 98.25) - open/close 1959 (time 1763344088.29 total 20.01 last 97.51) - open/close 2988 (time 1763344098.30 total 30.02 last 102.82) - open/close 4018 (time 1763344108.32 total 40.04 last 102.85) - open/close 4947 (time 1763344118.33 total 50.05 last 92.79) - open/close 5878 (time 1763344128.33 total 60.05 last 93.07) - open/close 6803 (time 1763344138.34 total 70.06 last 92.41) - open/close 7764 (time 1763344148.35 total 80.07 last 96.05) - open/close 8723 (time 1763344158.35 total 90.07 last 95.87) - open/close 9662 (time 1763344168.36 total 100.08 last 93.79) total: 10000 open/close in 103.71 seconds: 96.43 ops/second - unlinked 0 (time 1763344175 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre On OST0, 20367 used inodes rc=0 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2