== conf-sanity test 69: replace an OST with the same index ========================================================== 04:14:18 (1768554858) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 670 (time 1768554991.31 total 10.03 last 66.77) - open/close 1267 (time 1768555001.32 total 20.04 last 59.68) - open/close 1977 (time 1768555011.33 total 30.05 last 70.91) - open/close 2557 (time 1768555021.35 total 40.07 last 57.89) - open/close 3153 (time 1768555031.36 total 50.09 last 59.50) - open/close 3825 (time 1768555041.37 total 60.09 last 67.16) - open/close 4481 (time 1768555051.38 total 70.10 last 65.52) - open/close 5214 (time 1768555061.39 total 80.11 last 73.23) - open/close 5870 (time 1768555071.40 total 90.12 last 65.54) - open/close 6641 (time 1768555081.40 total 100.12 last 77.09) - open/close 7490 (time 1768555091.40 total 110.13 last 84.87) - open/close 8154 (time 1768555101.41 total 120.14 last 66.34) - open/close 8940 (time 1768555111.42 total 130.14 last 78.55) - open/close 9752 (time 1768555121.44 total 140.16 last 81.05) total: 10000 open/close in 143.80 seconds: 69.54 ops/second - unlinked 0 (time 1768555130 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 722 (time 1768555225.99 total 10.01 last 72.12) - open/close 1486 (time 1768555236.00 total 20.01 last 76.38) - open/close 2301 (time 1768555246.00 total 30.02 last 81.44) - open/close 3098 (time 1768555256.01 total 40.02 last 79.67) - open/close 3871 (time 1768555266.01 total 50.03 last 77.25) - open/close 4628 (time 1768555276.02 total 60.03 last 75.69) - open/close 5336 (time 1768555286.02 total 70.04 last 70.75) - open/close 6088 (time 1768555296.04 total 80.06 last 75.07) - open/close 6743 (time 1768555306.05 total 90.06 last 65.45) - open/close 7401 (time 1768555316.07 total 100.09 last 65.64) - open/close 8241 (time 1768555326.08 total 110.10 last 83.90) - open/close 8959 (time 1768555336.09 total 120.10 last 71.79) - open/close 9607 (time 1768555346.09 total 130.11 last 64.77) total: 10000 open/close in 138.71 seconds: 72.09 ops/second - unlinked 0 (time 1768555360 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 615 (time 1768555460.92 total 10.01 last 61.41) - open/close 1356 (time 1768555470.93 total 20.03 last 74.01) - open/close 2122 (time 1768555480.96 total 30.05 last 76.40) - open/close 2974 (time 1768555490.96 total 40.05 last 85.19) - open/close 3729 (time 1768555500.97 total 50.07 last 75.41) - open/close 4389 (time 1768555510.97 total 60.07 last 65.98) - open/close 5016 (time 1768555520.98 total 70.08 last 62.63) - open/close 5773 (time 1768555530.99 total 80.08 last 75.66) - open/close 6359 (time 1768555541.01 total 90.11 last 58.46) - open/close 7150 (time 1768555551.04 total 100.13 last 78.92) - open/close 8053 (time 1768555561.05 total 110.14 last 90.19) - open/close 8800 (time 1768555571.06 total 120.15 last 74.65) - open/close 9555 (time 1768555581.06 total 130.15 last 75.50) total: 10000 open/close in 136.99 seconds: 73.00 ops/second - unlinked 0 (time 1768555592 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 587 (time 1768555694.37 total 10.01 last 58.64) - open/close 1244 (time 1768555704.40 total 20.03 last 65.54) - open/close 2035 (time 1768555714.40 total 30.04 last 79.04) - open/close 2818 (time 1768555724.42 total 40.06 last 78.14) - open/close 3550 (time 1768555734.42 total 50.06 last 73.20) - open/close 4273 (time 1768555744.44 total 60.07 last 72.21) - open/close 5104 (time 1768555754.44 total 70.08 last 83.08) - open/close 5835 (time 1768555764.45 total 80.08 last 73.06) - open/close 6496 (time 1768555774.45 total 90.09 last 66.07) - open/close 7153 (time 1768555784.46 total 100.09 last 65.66) - open/close 8020 (time 1768555794.46 total 110.10 last 86.66) - open/close 8682 (time 1768555804.47 total 120.10 last 66.16) - open/close 9516 (time 1768555814.47 total 130.11 last 83.39) total: 10000 open/close in 136.59 seconds: 73.21 ops/second - unlinked 0 (time 1768555825 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 700 (time 1768555916.59 total 10.00 last 69.98) - open/close 1495 (time 1768555926.59 total 20.01 last 79.48) - open/close 2221 (time 1768555936.60 total 30.01 last 72.55) - open/close 3019 (time 1768555946.62 total 40.03 last 79.66) - open/close 3726 (time 1768555956.62 total 50.04 last 70.66) - open/close 4543 (time 1768555966.64 total 60.05 last 81.60) - open/close 5356 (time 1768555976.65 total 70.06 last 81.21) - open/close 6263 (time 1768555986.65 total 80.06 last 90.69) - open/close 6949 (time 1768555996.67 total 90.08 last 68.48) - open/close 7657 (time 1768556006.67 total 100.08 last 70.78) - open/close 8455 (time 1768556016.67 total 110.08 last 79.80) - open/close 9229 (time 1768556026.68 total 120.09 last 77.35) - open/close 9979 (time 1768556036.68 total 130.09 last 74.99) total: 10000 open/close in 130.39 seconds: 76.69 ops/second - unlinked 0 (time 1768556042 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 809 (time 1768556140.34 total 10.01 last 80.80) - open/close 1510 (time 1768556150.34 total 20.01 last 70.09) - open/close 2350 (time 1768556160.34 total 30.01 last 84.00) - open/close 3217 (time 1768556170.35 total 40.02 last 86.63) - open/close 3972 (time 1768556180.36 total 50.03 last 75.43) - open/close 4849 (time 1768556190.36 total 60.04 last 87.65) - open/close 5674 (time 1768556200.37 total 70.04 last 82.47) - open/close 6617 (time 1768556210.37 total 80.04 last 94.30) - open/close 7396 (time 1768556220.37 total 90.05 last 77.86) - open/close 8146 (time 1768556230.37 total 100.05 last 74.98) - open/close 8896 (time 1768556240.40 total 110.08 last 74.81) - open/close 9651 (time 1768556250.40 total 120.08 last 75.46) total: 10000 open/close in 124.10 seconds: 80.58 ops/second - unlinked 0 (time 1768556258 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 908 (time 1768556347.68 total 10.00 last 90.76) - open/close 1691 (time 1768556357.69 total 20.02 last 78.19) - open/close 2482 (time 1768556367.69 total 30.02 last 79.09) - open/close 3214 (time 1768556377.69 total 40.02 last 73.20) - open/close 4021 (time 1768556387.70 total 50.03 last 80.64) - open/close 4871 (time 1768556397.71 total 60.04 last 84.93) - open/close 5697 (time 1768556407.72 total 70.05 last 82.49) - open/close 6662 (time 1768556417.74 total 80.07 last 96.30) - open/close 7471 (time 1768556427.75 total 90.07 last 80.87) - open/close 8288 (time 1768556437.76 total 100.08 last 81.64) - open/close 8999 (time 1768556447.78 total 110.11 last 70.92) - open/close 9781 (time 1768556457.78 total 120.11 last 78.20) total: 10000 open/close in 123.04 seconds: 81.27 ops/second - unlinked 0 (time 1768556465 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 855 (time 1768556555.96 total 10.00 last 85.48) - open/close 1668 (time 1768556565.96 total 20.01 last 81.27) - open/close 2474 (time 1768556575.96 total 30.01 last 80.58) - open/close 3293 (time 1768556585.97 total 40.01 last 81.87) - open/close 4040 (time 1768556595.97 total 50.01 last 74.69) - open/close 4892 (time 1768556605.98 total 60.02 last 85.14) - open/close 5619 (time 1768556615.98 total 70.02 last 72.67) - open/close 6434 (time 1768556625.98 total 80.03 last 81.46) - open/close 7193 (time 1768556635.99 total 90.03 last 75.86) - open/close 7985 (time 1768556645.99 total 100.03 last 79.19) - open/close 8793 (time 1768556656.00 total 110.05 last 80.70) - open/close 9457 (time 1768556666.01 total 120.06 last 66.32) total: 10000 open/close in 128.21 seconds: 78.00 ops/second - unlinked 0 (time 1768556678 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 729 (time 1768556771.44 total 10.00 last 72.88) - open/close 1436 (time 1768556781.46 total 20.02 last 70.55) - open/close 2164 (time 1768556791.47 total 30.03 last 72.73) - open/close 3037 (time 1768556801.48 total 40.04 last 87.26) - open/close 3763 (time 1768556811.48 total 50.04 last 72.57) - open/close 4479 (time 1768556821.49 total 60.05 last 71.55) - open/close 5352 (time 1768556831.49 total 70.05 last 87.30) - open/close 6103 (time 1768556841.50 total 80.06 last 75.01) - open/close 6930 (time 1768556851.51 total 90.07 last 82.63) - open/close 7708 (time 1768556861.51 total 100.07 last 77.78) - open/close 8433 (time 1768556871.52 total 110.08 last 72.44) - open/close 9186 (time 1768556881.54 total 120.10 last 75.16) - open/close 9935 (time 1768556891.54 total 130.10 last 74.87) total: 10000 open/close in 130.69 seconds: 76.52 ops/second - unlinked 0 (time 1768556897 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 711 (time 1768556990.46 total 10.01 last 71.04) - open/close 1454 (time 1768557000.47 total 20.02 last 74.20) - open/close 2242 (time 1768557010.48 total 30.03 last 78.74) - open/close 3008 (time 1768557020.48 total 40.04 last 76.55) - open/close 3812 (time 1768557030.49 total 50.04 last 80.39) - open/close 4488 (time 1768557040.49 total 60.04 last 67.58) - open/close 5296 (time 1768557050.49 total 70.04 last 80.76) - open/close 6177 (time 1768557060.51 total 80.06 last 87.93) - open/close 7041 (time 1768557070.51 total 90.07 last 86.38) - open/close 7806 (time 1768557080.53 total 100.08 last 76.37) - open/close 8607 (time 1768557090.54 total 110.09 last 80.05) - open/close 9344 (time 1768557100.55 total 120.10 last 73.64) total: 10000 open/close in 129.69 seconds: 77.11 ops/second - unlinked 0 (time 1768557115 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.110@tcp sys.timeout=20 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg410-server: oleg410-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg410-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg410-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2