== conf-sanity test 69: replace an OST with the same index ========================================================== 03:24:53 (1785309893) start mds service on oleg426-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing load_modules_local oleg426-server: Loading modules from /home/green/git/lustre-release/lustre oleg426-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg426-server: detected 4 online CPUs by sysfs oleg426-server: MODOPTS_LIBCFS= oleg426-server: Force libcfs to create 2 CPU partitions oleg426-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg426-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg426-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg426-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 578 (time 1785310043.13 total 10.00 last 57.78) - open/close 848 (time 1785310053.16 total 20.03 last 26.92) - open/close 1306 (time 1785310063.17 total 30.04 last 45.76) - open/close 1817 (time 1785310073.24 total 40.11 last 50.74) - open/close 2363 (time 1785310083.24 total 50.11 last 54.58) - open/close 2890 (time 1785310093.25 total 60.12 last 52.67) - open/close 3409 (time 1785310103.25 total 70.12 last 51.88) - open/close 3865 (time 1785310113.28 total 80.15 last 45.47) - open/close 4338 (time 1785310123.29 total 90.16 last 47.24) - open/close 4905 (time 1785310133.30 total 100.17 last 56.64) - open/close 5617 (time 1785310143.34 total 110.21 last 70.94) - open/close 6414 (time 1785310153.35 total 120.22 last 79.60) - open/close 7040 (time 1785310163.35 total 130.22 last 62.59) - open/close 7973 (time 1785310173.36 total 140.23 last 93.25) - open/close 8788 (time 1785310183.36 total 150.24 last 81.43) - open/close 9379 (time 1785310193.37 total 160.24 last 59.09) - open/close 9819 (time 1785310203.38 total 170.26 last 43.92) total: 10000 open/close in 174.86 seconds: 57.19 ops/second - unlinked 0 (time 1785310216 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 559 (time 1785310328.55 total 10.00 last 55.88) - open/close 1026 (time 1785310338.55 total 20.00 last 46.69) - open/close 1419 (time 1785310348.57 total 30.03 last 39.21) - open/close 1958 (time 1785310358.58 total 40.04 last 53.82) - open/close 2449 (time 1785310368.60 total 50.06 last 49.02) - open/close 2820 (time 1785310378.61 total 60.07 last 37.07) - open/close 3301 (time 1785310388.62 total 70.07 last 48.06) - open/close 3682 (time 1785310398.62 total 80.08 last 38.10) - open/close 4228 (time 1785310408.63 total 90.08 last 54.56) - open/close 4664 (time 1785310418.63 total 100.09 last 43.58) - open/close 5177 (time 1785310428.64 total 110.10 last 51.25) - open/close 5964 (time 1785310438.66 total 120.11 last 78.58) - open/close 6789 (time 1785310448.66 total 130.12 last 82.45) - open/close 7370 (time 1785310458.66 total 140.12 last 58.08) - open/close 7974 (time 1785310468.67 total 150.12 last 60.38) - open/close 8398 (time 1785310478.70 total 160.15 last 42.28) - open/close 9014 (time 1785310488.70 total 170.16 last 61.57) - open/close 9543 (time 1785310498.70 total 180.16 last 52.89) total: 10000 open/close in 185.49 seconds: 53.91 ops/second - unlinked 0 (time 1785310509 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 619 (time 1785310615.23 total 10.01 last 61.83) - open/close 1506 (time 1785310625.23 total 20.02 last 88.65) - open/close 2254 (time 1785310635.23 total 30.02 last 74.78) - open/close 2650 (time 1785310645.25 total 40.04 last 39.54) - open/close 3239 (time 1785310655.26 total 50.05 last 58.83) - open/close 4129 (time 1785310665.27 total 60.05 last 88.96) - open/close 4861 (time 1785310675.29 total 70.07 last 73.03) - open/close 5335 (time 1785310685.31 total 80.10 last 47.30) - open/close 6146 (time 1785310695.31 total 90.10 last 81.06) - open/close 6763 (time 1785310705.32 total 100.10 last 61.69) - open/close 7419 (time 1785310715.33 total 110.11 last 65.54) - open/close 8024 (time 1785310725.33 total 120.11 last 60.48) - open/close 8527 (time 1785310735.34 total 130.12 last 50.27) - open/close 9120 (time 1785310745.34 total 140.12 last 59.29) - open/close 9662 (time 1785310755.34 total 150.12 last 54.19) total: 10000 open/close in 157.13 seconds: 63.64 ops/second - unlinked 0 (time 1785310767 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 688 (time 1785310870.66 total 10.01 last 68.76) - open/close 1152 (time 1785310880.68 total 20.03 last 46.30) - open/close 1650 (time 1785310890.68 total 30.03 last 49.79) - open/close 2354 (time 1785310900.72 total 40.06 last 70.15) - open/close 3001 (time 1785310910.73 total 50.08 last 64.60) - open/close 3535 (time 1785310920.73 total 60.08 last 53.40) - open/close 4151 (time 1785310930.74 total 70.09 last 61.54) - open/close 4641 (time 1785310940.78 total 80.13 last 48.81) - open/close 5397 (time 1785310950.80 total 90.14 last 75.49) - open/close 5863 (time 1785310960.81 total 100.16 last 46.52) - open/close 6424 (time 1785310970.81 total 110.16 last 56.09) - open/close 7276 (time 1785310980.82 total 120.16 last 85.19) - open/close 7933 (time 1785310990.83 total 130.18 last 65.59) - open/close 8521 (time 1785311000.85 total 140.20 last 58.69) - open/close 9048 (time 1785311010.86 total 150.20 last 52.68) - open/close 9664 (time 1785311020.86 total 160.21 last 61.57) total: 10000 open/close in 166.29 seconds: 60.14 ops/second - unlinked 0 (time 1785311033 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 711 (time 1785311137.90 total 10.01 last 71.00) - open/close 1196 (time 1785311147.92 total 20.04 last 48.39) - open/close 1843 (time 1785311157.92 total 30.04 last 64.69) - open/close 2833 (time 1785311167.93 total 40.04 last 98.95) - open/close 3552 (time 1785311177.93 total 50.05 last 71.86) - open/close 4091 (time 1785311187.93 total 60.05 last 53.88) - open/close 4621 (time 1785311197.94 total 70.06 last 52.97) - open/close 5516 (time 1785311207.95 total 80.07 last 89.36) - open/close 6683 (time 1785311217.96 total 90.08 last 116.67) - open/close 7355 (time 1785311227.96 total 100.08 last 67.15) - open/close 7900 (time 1785311237.99 total 110.11 last 54.36) - open/close 8679 (time 1785311247.99 total 120.11 last 77.89) - open/close 9645 (time 1785311258.00 total 130.12 last 96.53) total: 10000 open/close in 135.66 seconds: 73.71 ops/second - unlinked 0 (time 1785311268 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 494 (time 1785311370.48 total 10.01 last 49.33) - open/close 999 (time 1785311380.49 total 20.02 last 50.48) - open/close 1581 (time 1785311390.49 total 30.02 last 58.19) - open/close 2389 (time 1785311400.49 total 40.03 last 80.74) - open/close 2971 (time 1785311410.50 total 50.04 last 58.16) - open/close 3796 (time 1785311420.51 total 60.04 last 82.43) - open/close 4426 (time 1785311430.53 total 70.06 last 62.88) - open/close 4995 (time 1785311440.53 total 80.07 last 56.88) - open/close 5720 (time 1785311450.54 total 90.07 last 72.46) - open/close 6180 (time 1785311460.55 total 100.08 last 45.97) - open/close 6774 (time 1785311470.56 total 110.09 last 59.33) - open/close 7363 (time 1785311480.57 total 120.11 last 58.81) - open/close 8378 (time 1785311490.58 total 130.11 last 101.46) - open/close 8893 (time 1785311500.58 total 140.12 last 51.48) - open/close 9508 (time 1785311510.59 total 150.13 last 61.42) total: 10000 open/close in 160.11 seconds: 62.46 ops/second - unlinked 0 (time 1785311525 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 566 (time 1785311616.77 total 10.02 last 56.48) - open/close 1279 (time 1785311626.77 total 20.02 last 71.28) - open/close 1894 (time 1785311636.77 total 30.02 last 61.50) - open/close 2470 (time 1785311646.78 total 40.03 last 57.55) - open/close 3101 (time 1785311656.80 total 50.05 last 63.00) - open/close 3929 (time 1785311666.80 total 60.05 last 82.78) - open/close 4738 (time 1785311676.81 total 70.06 last 80.80) - open/close 5635 (time 1785311686.82 total 80.07 last 89.63) - open/close 6899 (time 1785311696.83 total 90.08 last 126.32) - open/close 7660 (time 1785311706.84 total 100.09 last 76.04) - open/close 8312 (time 1785311716.85 total 110.10 last 65.14) - open/close 9309 (time 1785311726.85 total 120.10 last 99.70) - open/close 9901 (time 1785311736.86 total 130.11 last 59.12) total: 10000 open/close in 131.43 seconds: 76.09 ops/second - unlinked 0 (time 1785311742 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1195 (time 1785311837.74 total 10.01 last 119.43) - open/close 2003 (time 1785311847.75 total 20.02 last 80.71) - open/close 2770 (time 1785311857.75 total 30.02 last 76.68) - open/close 3590 (time 1785311867.75 total 40.02 last 82.00) - open/close 4369 (time 1785311877.75 total 50.02 last 77.89) - open/close 5128 (time 1785311887.76 total 60.03 last 75.85) - open/close 5687 (time 1785311897.76 total 70.03 last 55.89) - open/close 6331 (time 1785311907.78 total 80.05 last 64.30) - open/close 6852 (time 1785311917.79 total 90.06 last 52.02) - open/close 7725 (time 1785311927.79 total 100.06 last 87.28) - open/close 8470 (time 1785311937.79 total 110.06 last 74.49) - open/close 9092 (time 1785311947.80 total 120.07 last 62.19) - open/close 9674 (time 1785311957.80 total 130.07 last 58.20) total: 10000 open/close in 134.10 seconds: 74.57 ops/second - unlinked 0 (time 1785311966 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 956 (time 1785312060.70 total 10.00 last 95.57) - open/close 1702 (time 1785312070.71 total 20.01 last 74.53) - open/close 2323 (time 1785312080.72 total 30.03 last 62.01) - open/close 2976 (time 1785312090.73 total 40.04 last 65.22) - open/close 3622 (time 1785312100.74 total 50.05 last 64.55) - open/close 4318 (time 1785312110.76 total 60.07 last 69.47) - open/close 5012 (time 1785312120.76 total 70.07 last 69.38) - open/close 5901 (time 1785312130.77 total 80.08 last 88.83) - open/close 6658 (time 1785312140.78 total 90.08 last 75.64) - open/close 7319 (time 1785312150.78 total 100.09 last 66.10) - open/close 8055 (time 1785312160.79 total 110.10 last 73.49) - open/close 8723 (time 1785312170.79 total 120.10 last 66.80) - open/close 9596 (time 1785312180.80 total 130.11 last 87.25) total: 10000 open/close in 134.24 seconds: 74.49 ops/second - unlinked 0 (time 1785312189 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 713 (time 1785312267.69 total 10.00 last 71.29) - open/close 1433 (time 1785312277.69 total 20.00 last 71.98) - open/close 2135 (time 1785312287.70 total 30.01 last 70.16) - open/close 2729 (time 1785312297.70 total 40.01 last 59.38) - open/close 3718 (time 1785312307.70 total 50.01 last 98.88) - open/close 4280 (time 1785312317.70 total 60.02 last 56.18) - open/close 4888 (time 1785312327.71 total 70.03 last 60.75) - open/close 5549 (time 1785312337.74 total 80.05 last 65.94) - open/close 6212 (time 1785312347.75 total 90.06 last 66.24) - open/close 6965 (time 1785312357.75 total 100.06 last 75.30) - open/close 7739 (time 1785312367.77 total 110.08 last 77.25) - open/close 8315 (time 1785312377.77 total 120.08 last 57.59) - open/close 8930 (time 1785312387.77 total 130.09 last 61.46) - open/close 9597 (time 1785312397.80 total 140.11 last 66.55) total: 10000 open/close in 146.54 seconds: 68.24 ops/second - unlinked 0 (time 1785312408 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.126@tcp sys.timeout=20 start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg426-server: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg426-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg426-server unloading modules via unload_modules_local on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing unload_modules_local oleg426-server: modules unloaded. pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2