== conf-sanity test 69: replace an OST with the same index ========================================================== 16:25:48 (1785788748) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 654 (time 1785788887.55 total 10.00 last 65.38) - open/close 1436 (time 1785788897.56 total 20.01 last 78.14) - open/close 2087 (time 1785788907.59 total 30.04 last 64.92) - open/close 2939 (time 1785788917.59 total 40.04 last 85.19) - open/close 3614 (time 1785788927.60 total 50.05 last 67.42) - open/close 4203 (time 1785788937.62 total 60.07 last 58.80) - open/close 4686 (time 1785788947.64 total 70.09 last 48.21) - open/close 5210 (time 1785788957.65 total 80.10 last 52.33) - open/close 5942 (time 1785788967.66 total 90.11 last 73.12) - open/close 6492 (time 1785788977.68 total 100.13 last 54.91) - open/close 7004 (time 1785788987.68 total 110.13 last 51.19) - open/close 7577 (time 1785788997.69 total 120.14 last 57.27) - open/close 8149 (time 1785789007.70 total 130.15 last 57.12) - open/close 8748 (time 1785789017.71 total 140.16 last 59.87) - open/close 9195 (time 1785789027.74 total 150.19 last 44.54) - open/close 9709 (time 1785789037.77 total 160.22 last 51.25) total: 10000 open/close in 166.20 seconds: 60.17 ops/second - unlinked 0 (time 1785789049 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 561 (time 1785789151.70 total 10.00 last 56.09) - open/close 1111 (time 1785789161.72 total 20.02 last 54.91) - open/close 2129 (time 1785789171.72 total 30.02 last 101.78) - open/close 2893 (time 1785789181.72 total 40.02 last 76.38) - open/close 3993 (time 1785789191.74 total 50.03 last 109.88) - open/close 4863 (time 1785789201.74 total 60.04 last 86.95) - open/close 5516 (time 1785789211.75 total 70.05 last 65.23) - open/close 6153 (time 1785789221.76 total 80.06 last 63.64) - open/close 6856 (time 1785789231.76 total 90.06 last 70.28) - open/close 7421 (time 1785789241.77 total 100.07 last 56.46) - open/close 7988 (time 1785789251.79 total 110.09 last 56.59) - open/close 8839 (time 1785789261.79 total 120.09 last 85.09) - open/close 9321 (time 1785789271.81 total 130.11 last 48.11) - open/close 9880 (time 1785789281.81 total 140.11 last 55.89) total: 10000 open/close in 142.72 seconds: 70.07 ops/second - unlinked 0 (time 1785789289 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 640 (time 1785789377.88 total 10.00 last 63.98) - open/close 1330 (time 1785789387.88 total 20.01 last 68.96) - open/close 2029 (time 1785789397.89 total 30.01 last 69.88) - open/close 2873 (time 1785789407.91 total 40.03 last 84.24) - open/close 3768 (time 1785789417.91 total 50.04 last 89.45) - open/close 4467 (time 1785789427.91 total 60.04 last 69.89) - open/close 5033 (time 1785789437.93 total 70.06 last 56.47) - open/close 5701 (time 1785789447.94 total 80.06 last 66.79) - open/close 6219 (time 1785789457.95 total 90.08 last 51.71) - open/close 6792 (time 1785789467.96 total 100.08 last 57.26) - open/close 7309 (time 1785789477.96 total 110.09 last 51.68) - open/close 7874 (time 1785789487.98 total 120.10 last 56.41) - open/close 8497 (time 1785789497.98 total 130.10 last 62.30) - open/close 9070 (time 1785789507.99 total 140.12 last 57.23) - open/close 9744 (time 1785789518.00 total 150.12 last 67.36) total: 10000 open/close in 154.30 seconds: 64.81 ops/second - unlinked 0 (time 1785789527 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 627 (time 1785789628.88 total 10.00 last 62.67) - open/close 1422 (time 1785789638.89 total 20.01 last 79.47) - open/close 2071 (time 1785789648.89 total 30.01 last 64.88) - open/close 2676 (time 1785789658.90 total 40.03 last 60.41) - open/close 3243 (time 1785789668.91 total 50.04 last 56.65) - open/close 3759 (time 1785789678.93 total 60.05 last 51.51) - open/close 4375 (time 1785789688.93 total 70.06 last 61.57) - open/close 5092 (time 1785789698.95 total 80.08 last 71.56) - open/close 5656 (time 1785789708.96 total 90.08 last 56.37) - open/close 6715 (time 1785789718.96 total 100.09 last 105.84) - open/close 7443 (time 1785789728.97 total 110.09 last 72.75) - open/close 8035 (time 1785789738.98 total 120.11 last 59.12) - open/close 8611 (time 1785789749.00 total 130.12 last 57.52) - open/close 9315 (time 1785789759.00 total 140.12 last 70.40) - open/close 9864 (time 1785789769.00 total 150.12 last 54.88) total: 10000 open/close in 152.63 seconds: 65.52 ops/second - unlinked 0 (time 1785789776 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 570 (time 1785789866.05 total 10.01 last 56.96) - open/close 1174 (time 1785789876.06 total 20.01 last 60.36) - open/close 1712 (time 1785789886.07 total 30.02 last 53.74) - open/close 2443 (time 1785789896.07 total 40.03 last 73.07) - open/close 2976 (time 1785789906.08 total 50.03 last 53.28) - open/close 3537 (time 1785789916.09 total 60.04 last 56.05) - open/close 4360 (time 1785789926.09 total 70.05 last 82.25) - open/close 5242 (time 1785789936.13 total 80.08 last 87.88) - open/close 5954 (time 1785789946.13 total 90.09 last 71.19) - open/close 6430 (time 1785789956.14 total 100.09 last 47.56) - open/close 6997 (time 1785789966.16 total 110.12 last 56.58) - open/close 7833 (time 1785789976.17 total 120.12 last 83.53) - open/close 8503 (time 1785789986.18 total 130.14 last 66.93) - open/close 9592 (time 1785789996.18 total 140.14 last 108.85) total: 10000 open/close in 144.89 seconds: 69.02 ops/second - unlinked 0 (time 1785790005 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 657 (time 1785790103.81 total 10.00 last 65.69) - open/close 1214 (time 1785790113.82 total 20.01 last 55.68) - open/close 1851 (time 1785790123.94 total 30.13 last 62.93) - open/close 2392 (time 1785790133.94 total 40.13 last 54.09) - open/close 2895 (time 1785790143.96 total 50.15 last 50.20) - open/close 3459 (time 1785790153.97 total 60.16 last 56.32) - open/close 3935 (time 1785790164.00 total 70.19 last 47.48) - open/close 4486 (time 1785790174.01 total 80.20 last 55.06) - open/close 5085 (time 1785790184.03 total 90.22 last 59.74) - open/close 5843 (time 1785790194.04 total 100.23 last 75.72) - open/close 6868 (time 1785790204.05 total 110.24 last 102.46) - open/close 7540 (time 1785790214.06 total 120.25 last 67.15) - open/close 8299 (time 1785790224.06 total 130.25 last 75.86) - open/close 8910 (time 1785790234.06 total 140.25 last 61.08) - open/close 9556 (time 1785790244.06 total 150.25 last 64.59) total: 10000 open/close in 158.52 seconds: 63.08 ops/second - unlinked 0 (time 1785790257 ; total 0 ; last 0) total: 10000 unlinks in 130 seconds: 76.923080 unlinks/second - open/close 486 (time 1785790403.23 total 10.04 last 48.40) - open/close 948 (time 1785790413.25 total 20.06 last 46.11) - open/close 1443 (time 1785790423.27 total 30.08 last 49.40) - open/close 1959 (time 1785790433.28 total 40.09 last 51.57) - open/close 2662 (time 1785790443.28 total 50.09 last 70.28) - open/close 3306 (time 1785790453.29 total 60.10 last 64.36) - open/close 3866 (time 1785790463.29 total 70.10 last 55.99) - open/close 4428 (time 1785790473.31 total 80.12 last 56.05) - open/close 5125 (time 1785790483.33 total 90.14 last 69.57) - open/close 5587 (time 1785790493.33 total 100.15 last 46.18) - open/close 6103 (time 1785790503.35 total 110.16 last 51.55) - open/close 6770 (time 1785790513.37 total 120.18 last 66.56) - open/close 7372 (time 1785790523.37 total 130.18 last 60.19) - open/close 7820 (time 1785790533.37 total 140.18 last 44.80) - open/close 8747 (time 1785790543.37 total 150.18 last 92.70) - open/close 9666 (time 1785790553.37 total 160.19 last 91.84) total: 10000 open/close in 163.82 seconds: 61.04 ops/second - unlinked 0 (time 1785790562 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 551 (time 1785790666.92 total 10.03 last 54.94) - open/close 1016 (time 1785790676.93 total 20.04 last 46.47) - open/close 1659 (time 1785790686.94 total 30.05 last 64.21) - open/close 2405 (time 1785790696.95 total 40.05 last 74.57) - open/close 2888 (time 1785790706.96 total 50.07 last 48.24) - open/close 3586 (time 1785790716.96 total 60.07 last 69.78) - open/close 4362 (time 1785790726.98 total 70.09 last 77.47) - open/close 5011 (time 1785790736.99 total 80.09 last 64.85) - open/close 5777 (time 1785790747.01 total 90.12 last 76.38) - open/close 6414 (time 1785790757.02 total 100.13 last 63.67) - open/close 7130 (time 1785790767.05 total 110.16 last 71.41) - open/close 7627 (time 1785790777.07 total 120.18 last 49.56) - open/close 8248 (time 1785790787.08 total 130.19 last 62.05) - open/close 9043 (time 1785790797.10 total 140.21 last 79.39) - open/close 9580 (time 1785790807.10 total 150.21 last 53.68) total: 10000 open/close in 157.75 seconds: 63.39 ops/second - unlinked 0 (time 1785790820 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 574 (time 1785790915.83 total 10.01 last 57.32) - open/close 1120 (time 1785790925.84 total 20.02 last 54.56) - open/close 1873 (time 1785790935.84 total 30.02 last 75.29) - open/close 2482 (time 1785790945.86 total 40.05 last 60.76) - open/close 3136 (time 1785790955.88 total 50.06 last 65.28) - open/close 3804 (time 1785790965.90 total 60.08 last 66.70) - open/close 4368 (time 1785790975.91 total 70.09 last 56.34) - open/close 4962 (time 1785790985.91 total 80.10 last 59.35) - open/close 5711 (time 1785790995.92 total 90.10 last 74.89) - open/close 6035 (time 1785791005.94 total 100.12 last 32.32) - open/close 6401 (time 1785791015.95 total 110.13 last 36.58) - open/close 6803 (time 1785791025.95 total 120.13 last 40.18) - open/close 7296 (time 1785791035.96 total 130.14 last 49.26) - open/close 7686 (time 1785791045.98 total 140.16 last 38.94) - open/close 8342 (time 1785791055.98 total 150.16 last 65.60) - open/close 8960 (time 1785791065.98 total 160.16 last 61.80) - open/close 9605 (time 1785791075.98 total 170.16 last 64.48) total: 10000 open/close in 178.05 seconds: 56.16 ops/second - unlinked 0 (time 1785791089 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 737 (time 1785791224.45 total 10.03 last 73.51) - open/close 1265 (time 1785791234.47 total 20.05 last 52.68) - open/close 1779 (time 1785791244.47 total 30.05 last 51.39) - open/close 2345 (time 1785791254.47 total 40.05 last 56.59) - open/close 3292 (time 1785791264.48 total 50.06 last 94.60) - open/close 4022 (time 1785791274.49 total 60.06 last 72.99) - open/close 4635 (time 1785791284.49 total 70.07 last 61.27) - open/close 5339 (time 1785791294.54 total 80.12 last 70.04) - open/close 5825 (time 1785791304.55 total 90.13 last 48.56) - open/close 6768 (time 1785791314.55 total 100.13 last 94.28) - open/close 7414 (time 1785791324.55 total 110.13 last 64.59) - open/close 7985 (time 1785791334.56 total 120.14 last 57.05) - open/close 8567 (time 1785791344.57 total 130.14 last 58.18) - open/close 9257 (time 1785791354.57 total 140.15 last 68.97) - open/close 9896 (time 1785791364.58 total 150.16 last 63.83) total: 10000 open/close in 153.44 seconds: 65.17 ops/second - unlinked 0 (time 1785791375 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.152@tcp sys.timeout=20 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2