== conf-sanity test 69: replace an OST with the same index ========================================================== 12:02:59 (1772470979) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg445-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 765 (time 1772471079.17 total 10.00 last 76.49) - open/close 1636 (time 1772471089.18 total 20.01 last 87.05) - open/close 2442 (time 1772471099.18 total 30.01 last 80.57) - open/close 3060 (time 1772471109.19 total 40.02 last 61.73) - open/close 3822 (time 1772471119.20 total 50.03 last 76.14) - open/close 4656 (time 1772471129.20 total 60.03 last 83.38) - open/close 5477 (time 1772471139.21 total 70.04 last 82.01) - open/close 6211 (time 1772471149.22 total 80.05 last 73.36) - open/close 7087 (time 1772471159.22 total 90.05 last 87.58) - open/close 7820 (time 1772471169.23 total 100.06 last 73.24) - open/close 8505 (time 1772471179.24 total 110.07 last 68.42) - open/close 9197 (time 1772471189.24 total 120.07 last 69.19) - open/close 9960 (time 1772471199.25 total 130.08 last 76.25) total: 10000 open/close in 130.63 seconds: 76.55 ops/second - unlinked 0 (time 1772471203 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 868 (time 1772471289.62 total 10.00 last 86.78) - open/close 1766 (time 1772471299.62 total 20.00 last 89.78) - open/close 2534 (time 1772471309.62 total 30.01 last 76.77) - open/close 3327 (time 1772471319.63 total 40.01 last 79.26) - open/close 4057 (time 1772471329.63 total 50.02 last 72.96) - open/close 4938 (time 1772471339.64 total 60.02 last 88.07) - open/close 5849 (time 1772471349.68 total 70.06 last 90.74) - open/close 6663 (time 1772471359.71 total 80.09 last 81.16) - open/close 7482 (time 1772471369.72 total 90.10 last 81.80) - open/close 8182 (time 1772471379.74 total 100.12 last 69.86) - open/close 8852 (time 1772471389.74 total 110.12 last 66.99) - open/close 9563 (time 1772471399.74 total 120.13 last 71.09) total: 10000 open/close in 125.17 seconds: 79.89 ops/second - unlinked 0 (time 1772471409 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 856 (time 1772471503.05 total 10.01 last 85.50) - open/close 1622 (time 1772471513.06 total 20.02 last 76.52) - open/close 2306 (time 1772471523.06 total 30.03 last 68.36) - open/close 2981 (time 1772471533.06 total 40.03 last 67.49) - open/close 3757 (time 1772471543.06 total 50.03 last 77.60) - open/close 4598 (time 1772471553.07 total 60.04 last 84.02) - open/close 5305 (time 1772471563.08 total 70.05 last 70.64) - open/close 5981 (time 1772471573.09 total 80.05 last 67.56) - open/close 6685 (time 1772471583.10 total 90.06 last 70.35) - open/close 7430 (time 1772471593.10 total 100.06 last 74.49) - open/close 8129 (time 1772471603.10 total 110.06 last 69.89) - open/close 8976 (time 1772471613.10 total 120.07 last 84.66) - open/close 9801 (time 1772471623.10 total 130.07 last 82.49) total: 10000 open/close in 132.91 seconds: 75.24 ops/second - unlinked 0 (time 1772471630 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 808 (time 1772471723.26 total 10.01 last 80.76) - open/close 1529 (time 1772471733.28 total 20.02 last 71.97) - open/close 2268 (time 1772471743.28 total 30.02 last 73.89) - open/close 3036 (time 1772471753.29 total 40.04 last 76.71) - open/close 3885 (time 1772471763.30 total 50.04 last 84.89) - open/close 4805 (time 1772471773.30 total 60.04 last 91.95) - open/close 5617 (time 1772471783.32 total 70.06 last 81.04) - open/close 6422 (time 1772471793.33 total 80.07 last 80.45) - open/close 7302 (time 1772471803.33 total 90.08 last 87.94) - open/close 8198 (time 1772471813.35 total 100.09 last 89.49) - open/close 8960 (time 1772471823.36 total 110.11 last 76.07) - open/close 9688 (time 1772471833.37 total 120.11 last 72.78) total: 10000 open/close in 124.17 seconds: 80.53 ops/second - unlinked 0 (time 1772471842 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 727 (time 1772471931.39 total 10.01 last 72.61) - open/close 1415 (time 1772471941.41 total 20.03 last 68.65) - open/close 2129 (time 1772471951.42 total 30.04 last 71.36) - open/close 2928 (time 1772471961.42 total 40.05 last 79.83) - open/close 3570 (time 1772471971.43 total 50.05 last 64.17) - open/close 4368 (time 1772471981.44 total 60.07 last 79.68) - open/close 5032 (time 1772471991.46 total 70.08 last 66.31) - open/close 5705 (time 1772472001.48 total 80.11 last 67.13) - open/close 6313 (time 1772472011.48 total 90.11 last 60.79) - open/close 7040 (time 1772472021.49 total 100.12 last 72.62) - open/close 7700 (time 1772472031.50 total 110.12 last 65.97) - open/close 8428 (time 1772472041.51 total 120.13 last 72.75) - open/close 9113 (time 1772472051.51 total 130.13 last 68.49) - open/close 9810 (time 1772472061.52 total 140.14 last 69.61) total: 10000 open/close in 142.30 seconds: 70.28 ops/second - unlinked 0 (time 1772472068 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 653 (time 1772472163.22 total 10.02 last 65.19) - open/close 1290 (time 1772472173.22 total 20.02 last 63.66) - open/close 1912 (time 1772472183.23 total 30.03 last 62.15) - open/close 2673 (time 1772472193.26 total 40.06 last 75.87) - open/close 3328 (time 1772472203.27 total 50.07 last 65.45) - open/close 4077 (time 1772472213.28 total 60.08 last 74.84) - open/close 4896 (time 1772472223.29 total 70.09 last 81.79) - open/close 5527 (time 1772472233.30 total 80.10 last 63.03) - open/close 6222 (time 1772472243.32 total 90.12 last 69.40) - open/close 6991 (time 1772472253.33 total 100.13 last 76.83) - open/close 7820 (time 1772472263.33 total 110.13 last 82.85) - open/close 8699 (time 1772472273.34 total 120.15 last 87.79) - open/close 9417 (time 1772472283.36 total 130.16 last 71.72) total: 10000 open/close in 138.71 seconds: 72.09 ops/second - unlinked 0 (time 1772472296 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 770 (time 1772472387.71 total 10.00 last 76.98) - open/close 1512 (time 1772472397.73 total 20.02 last 74.10) - open/close 2375 (time 1772472407.73 total 30.02 last 86.23) - open/close 3098 (time 1772472417.74 total 40.03 last 72.25) - open/close 3679 (time 1772472427.74 total 50.03 last 58.08) - open/close 4315 (time 1772472437.75 total 60.04 last 63.59) - open/close 4999 (time 1772472447.76 total 70.05 last 68.31) - open/close 5640 (time 1772472457.76 total 80.05 last 64.09) - open/close 6321 (time 1772472467.77 total 90.06 last 68.05) - open/close 7104 (time 1772472477.77 total 100.06 last 78.26) - open/close 7827 (time 1772472487.77 total 110.06 last 72.28) - open/close 8539 (time 1772472497.78 total 120.07 last 71.17) - open/close 9397 (time 1772472507.79 total 130.08 last 85.71) total: 10000 open/close in 137.90 seconds: 72.52 ops/second - unlinked 0 (time 1772472520 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 526 (time 1772472623.69 total 10.02 last 52.52) - open/close 1215 (time 1772472633.71 total 20.03 last 68.81) - open/close 1951 (time 1772472643.71 total 30.03 last 73.58) - open/close 2598 (time 1772472653.72 total 40.04 last 64.62) - open/close 3341 (time 1772472663.75 total 50.07 last 74.11) - open/close 3971 (time 1772472673.75 total 60.07 last 62.98) - open/close 4716 (time 1772472683.76 total 70.08 last 74.46) - open/close 5507 (time 1772472693.76 total 80.09 last 79.04) - open/close 6185 (time 1772472703.77 total 90.09 last 67.77) - open/close 6935 (time 1772472713.77 total 100.09 last 74.99) - open/close 7647 (time 1772472723.77 total 110.09 last 71.20) - open/close 8444 (time 1772472733.77 total 120.10 last 79.67) - open/close 9174 (time 1772472743.78 total 130.11 last 72.93) - open/close 9887 (time 1772472753.80 total 140.13 last 71.16) total: 10000 open/close in 141.81 seconds: 70.52 ops/second - unlinked 0 (time 1772472760 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 824 (time 1772472852.92 total 10.02 last 82.20) - open/close 1415 (time 1772472862.93 total 20.04 last 59.01) - open/close 1985 (time 1772472872.97 total 30.07 last 56.82) - open/close 2671 (time 1772472882.97 total 40.07 last 68.58) - open/close 3393 (time 1772472892.97 total 50.08 last 72.15) - open/close 3960 (time 1772472902.98 total 60.08 last 56.70) - open/close 4600 (time 1772472912.98 total 70.09 last 63.97) - open/close 5430 (time 1772472922.99 total 80.10 last 82.92) - open/close 6165 (time 1772472933.00 total 90.11 last 73.44) - open/close 6767 (time 1772472943.00 total 100.11 last 60.20) - open/close 7364 (time 1772472953.02 total 110.12 last 59.60) - open/close 8076 (time 1772472963.03 total 120.13 last 71.13) - open/close 8767 (time 1772472973.03 total 130.13 last 69.09) - open/close 9680 (time 1772472983.04 total 140.15 last 91.18) total: 10000 open/close in 145.61 seconds: 68.67 ops/second - unlinked 0 (time 1772472993 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 677 (time 1772473098.18 total 10.01 last 67.61) - open/close 1112 (time 1772473108.20 total 20.03 last 43.41) - open/close 1696 (time 1772473118.21 total 30.04 last 58.34) - open/close 2316 (time 1772473128.21 total 40.05 last 61.96) - open/close 3012 (time 1772473138.23 total 50.06 last 69.50) - open/close 3693 (time 1772473148.23 total 60.07 last 68.07) - open/close 4472 (time 1772473158.23 total 70.07 last 77.90) - open/close 5124 (time 1772473168.24 total 80.07 last 65.18) - open/close 5914 (time 1772473178.25 total 90.09 last 78.86) - open/close 6519 (time 1772473188.28 total 100.12 last 60.34) - open/close 7105 (time 1772473198.28 total 110.12 last 58.58) - open/close 7907 (time 1772473208.29 total 120.12 last 80.18) - open/close 8596 (time 1772473218.29 total 130.12 last 68.90) - open/close 9205 (time 1772473228.31 total 140.14 last 60.78) - open/close 9813 (time 1772473238.32 total 150.16 last 60.72) total: 10000 open/close in 153.39 seconds: 65.19 ops/second - unlinked 0 (time 1772473247 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.145@tcp sys.timeout=20 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2