== conf-sanity test 69: replace an OST with the same index ========================================================== 14:55:27 (1787856927) start mds service on oleg402-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg402-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 605 (time 1787857048.40 total 10.00 last 60.48) - open/close 1232 (time 1787857058.41 total 20.01 last 62.63) - open/close 1863 (time 1787857068.41 total 30.01 last 63.10) - open/close 2588 (time 1787857078.42 total 40.02 last 72.44) - open/close 3300 (time 1787857088.42 total 50.03 last 71.16) - open/close 4005 (time 1787857098.43 total 60.03 last 70.46) - open/close 4712 (time 1787857108.44 total 70.05 last 70.62) - open/close 5437 (time 1787857118.44 total 80.05 last 72.46) - open/close 6145 (time 1787857128.45 total 90.05 last 70.78) - open/close 6782 (time 1787857138.45 total 100.06 last 63.65) - open/close 7351 (time 1787857148.46 total 110.07 last 56.86) - open/close 8033 (time 1787857158.47 total 120.08 last 68.15) - open/close 8753 (time 1787857168.48 total 130.08 last 71.96) - open/close 9439 (time 1787857178.48 total 140.08 last 68.59) total: 10000 open/close in 147.88 seconds: 67.62 ops/second - unlinked 0 (time 1787857190 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 834 (time 1787857284.47 total 10.00 last 83.40) - open/close 1660 (time 1787857294.48 total 20.01 last 82.49) - open/close 2456 (time 1787857304.48 total 30.01 last 79.59) - open/close 3135 (time 1787857314.48 total 40.02 last 67.89) - open/close 3828 (time 1787857324.48 total 50.02 last 69.29) - open/close 4551 (time 1787857334.49 total 60.02 last 72.25) - open/close 5313 (time 1787857344.49 total 70.03 last 76.18) - open/close 6040 (time 1787857354.50 total 80.03 last 72.64) - open/close 6745 (time 1787857364.53 total 90.06 last 70.29) - open/close 7464 (time 1787857374.54 total 100.07 last 71.85) - open/close 8217 (time 1787857384.55 total 110.08 last 75.23) - open/close 8945 (time 1787857394.56 total 120.10 last 72.68) - open/close 9734 (time 1787857404.57 total 130.11 last 78.83) total: 10000 open/close in 133.44 seconds: 74.94 ops/second - unlinked 0 (time 1787857412 ; total 1 ; last 1) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 593 (time 1787857503.93 total 10.02 last 59.20) - open/close 1159 (time 1787857513.94 total 20.03 last 56.54) - open/close 1799 (time 1787857523.95 total 30.03 last 63.96) - open/close 2570 (time 1787857533.96 total 40.05 last 77.02) - open/close 3340 (time 1787857543.98 total 50.06 last 76.86) - open/close 4067 (time 1787857553.99 total 60.07 last 72.63) - open/close 4792 (time 1787857563.99 total 70.08 last 72.46) - open/close 5523 (time 1787857574.00 total 80.09 last 73.05) - open/close 6274 (time 1787857584.01 total 90.10 last 75.03) - open/close 7017 (time 1787857594.02 total 100.10 last 74.25) - open/close 7790 (time 1787857604.04 total 110.12 last 77.16) - open/close 8510 (time 1787857614.04 total 120.12 last 71.98) - open/close 9156 (time 1787857624.04 total 130.12 last 64.60) - open/close 9908 (time 1787857634.05 total 140.13 last 75.15) total: 10000 open/close in 141.47 seconds: 70.69 ops/second - unlinked 0 (time 1787857639 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 724 (time 1787857738.60 total 10.01 last 72.34) - open/close 1433 (time 1787857748.61 total 20.02 last 70.85) - open/close 2089 (time 1787857758.61 total 30.02 last 65.55) - open/close 2837 (time 1787857768.61 total 40.02 last 74.80) - open/close 3509 (time 1787857778.62 total 50.03 last 67.17) - open/close 4133 (time 1787857788.62 total 60.03 last 62.38) - open/close 4817 (time 1787857798.64 total 70.05 last 68.29) - open/close 5474 (time 1787857808.65 total 80.06 last 65.62) - open/close 6168 (time 1787857818.66 total 90.07 last 69.32) - open/close 6892 (time 1787857828.66 total 100.07 last 72.39) - open/close 7527 (time 1787857838.67 total 110.08 last 63.47) - open/close 8129 (time 1787857848.67 total 120.08 last 60.18) - open/close 8709 (time 1787857858.68 total 130.09 last 57.94) - open/close 9267 (time 1787857868.69 total 140.10 last 55.72) - open/close 9928 (time 1787857878.70 total 150.11 last 66.06) total: 10000 open/close in 151.13 seconds: 66.17 ops/second - unlinked 0 (time 1787857885 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 626 (time 1787857987.89 total 10.00 last 62.58) - open/close 1331 (time 1787857997.90 total 20.01 last 70.43) - open/close 2067 (time 1787858007.90 total 30.02 last 73.58) - open/close 2785 (time 1787858017.92 total 40.04 last 71.63) - open/close 3443 (time 1787858027.93 total 50.04 last 65.77) - open/close 4069 (time 1787858037.94 total 60.05 last 62.54) - open/close 4794 (time 1787858047.94 total 70.06 last 72.45) - open/close 5470 (time 1787858057.95 total 80.07 last 67.56) - open/close 6124 (time 1787858067.97 total 90.08 last 65.28) - open/close 6752 (time 1787858077.97 total 100.09 last 62.79) - open/close 7395 (time 1787858087.97 total 110.09 last 64.30) - open/close 7990 (time 1787858097.98 total 120.10 last 59.43) - open/close 8558 (time 1787858107.99 total 130.11 last 56.75) - open/close 9096 (time 1787858117.99 total 140.11 last 53.78) - open/close 9628 (time 1787858128.01 total 150.12 last 53.13) total: 10000 open/close in 156.03 seconds: 64.09 ops/second - unlinked 0 (time 1787858138 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 470 (time 1787858244.51 total 10.01 last 46.95) - open/close 978 (time 1787858254.52 total 20.02 last 50.74) - open/close 1548 (time 1787858264.52 total 30.03 last 56.97) - open/close 2313 (time 1787858274.53 total 40.03 last 76.47) - open/close 2964 (time 1787858284.53 total 50.04 last 65.05) - open/close 3656 (time 1787858294.54 total 60.05 last 69.15) - open/close 4313 (time 1787858304.54 total 70.05 last 65.70) - open/close 4930 (time 1787858314.55 total 80.05 last 61.65) - open/close 5540 (time 1787858324.57 total 90.08 last 60.85) - open/close 6344 (time 1787858334.57 total 100.08 last 80.40) - open/close 7008 (time 1787858344.58 total 110.08 last 66.38) - open/close 7654 (time 1787858354.58 total 120.09 last 64.55) - open/close 8351 (time 1787858364.59 total 130.09 last 69.66) - open/close 8955 (time 1787858374.59 total 140.09 last 60.40) - open/close 9577 (time 1787858384.59 total 150.09 last 62.20) total: 10000 open/close in 156.92 seconds: 63.73 ops/second - unlinked 0 (time 1787858397 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 662 (time 1787858501.49 total 10.01 last 66.12) - open/close 1329 (time 1787858511.50 total 20.03 last 66.61) - open/close 1916 (time 1787858521.51 total 30.04 last 58.64) - open/close 2557 (time 1787858531.54 total 40.06 last 63.95) - open/close 3109 (time 1787858541.54 total 50.07 last 55.16) - open/close 3651 (time 1787858551.55 total 60.08 last 54.15) - open/close 4251 (time 1787858561.56 total 70.08 last 59.96) - open/close 4938 (time 1787858571.56 total 80.08 last 68.70) - open/close 5624 (time 1787858581.57 total 90.10 last 68.52) - open/close 6310 (time 1787858591.58 total 100.10 last 68.54) - open/close 7008 (time 1787858601.59 total 110.11 last 69.74) - open/close 7747 (time 1787858611.59 total 120.12 last 73.86) - open/close 8412 (time 1787858621.60 total 130.13 last 66.44) - open/close 9103 (time 1787858631.61 total 140.13 last 69.08) - open/close 9829 (time 1787858641.61 total 150.13 last 72.59) total: 10000 open/close in 152.32 seconds: 65.65 ops/second - unlinked 0 (time 1787858648 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 607 (time 1787858745.83 total 10.02 last 60.61) - open/close 1264 (time 1787858755.86 total 20.05 last 65.47) - open/close 1891 (time 1787858765.88 total 30.07 last 62.58) - open/close 2509 (time 1787858775.89 total 40.08 last 61.76) - open/close 3168 (time 1787858785.90 total 50.09 last 65.83) - open/close 3904 (time 1787858795.92 total 60.11 last 73.46) - open/close 4650 (time 1787858805.93 total 70.12 last 74.54) - open/close 5343 (time 1787858815.93 total 80.12 last 69.26) - open/close 6124 (time 1787858825.93 total 90.12 last 78.09) - open/close 6829 (time 1787858835.94 total 100.13 last 70.47) - open/close 7549 (time 1787858845.94 total 110.13 last 72.00) - open/close 8304 (time 1787858855.95 total 120.14 last 75.40) - open/close 9055 (time 1787858865.96 total 130.15 last 75.05) - open/close 9798 (time 1787858875.97 total 140.16 last 74.23) total: 10000 open/close in 142.71 seconds: 70.07 ops/second - unlinked 0 (time 1787858882 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 483 (time 1787858975.39 total 10.04 last 48.12) - open/close 1081 (time 1787858985.40 total 20.05 last 59.72) - open/close 1732 (time 1787858995.41 total 30.05 last 65.08) - open/close 2404 (time 1787859005.41 total 40.06 last 67.15) - open/close 3082 (time 1787859015.42 total 50.07 last 67.75) - open/close 3649 (time 1787859025.43 total 60.08 last 56.65) - open/close 4224 (time 1787859035.43 total 70.08 last 57.50) - open/close 4965 (time 1787859045.44 total 80.08 last 74.04) - open/close 5718 (time 1787859055.46 total 90.10 last 75.17) - open/close 6430 (time 1787859065.46 total 100.11 last 71.16) - open/close 7032 (time 1787859075.47 total 110.12 last 60.14) - open/close 7678 (time 1787859085.48 total 120.12 last 64.57) - open/close 8370 (time 1787859095.50 total 130.15 last 69.02) - open/close 9032 (time 1787859105.50 total 140.15 last 66.18) - open/close 9698 (time 1787859115.51 total 150.15 last 66.59) total: 10000 open/close in 155.21 seconds: 64.43 ops/second - unlinked 0 (time 1787859125 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 801 (time 1787859222.03 total 10.00 last 80.06) - open/close 1593 (time 1787859232.03 total 20.01 last 79.17) - open/close 2425 (time 1787859242.03 total 30.01 last 83.18) - open/close 3164 (time 1787859252.05 total 40.02 last 73.82) - open/close 4000 (time 1787859262.05 total 50.02 last 83.60) - open/close 4762 (time 1787859272.05 total 60.03 last 76.16) - open/close 5513 (time 1787859282.07 total 70.04 last 74.99) - open/close 6194 (time 1787859292.07 total 80.05 last 68.05) - open/close 6885 (time 1787859302.08 total 90.06 last 69.05) - open/close 7668 (time 1787859312.09 total 100.07 last 78.22) - open/close 8478 (time 1787859322.09 total 110.07 last 80.98) - open/close 9260 (time 1787859332.09 total 120.07 last 78.19) total: 10000 open/close in 129.79 seconds: 77.05 ops/second - unlinked 0 (time 1787859346 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2