== conf-sanity test 69: replace an OST with the same index ========================================================== 18:46:46 (1753742806) start mds service on oleg119-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 782 (time 1753742899.38 total 10.01 last 78.13) - open/close 1661 (time 1753742909.39 total 20.01 last 87.88) - open/close 2426 (time 1753742919.39 total 30.02 last 76.44) - open/close 3315 (time 1753742929.40 total 40.02 last 88.89) - open/close 4134 (time 1753742939.41 total 50.03 last 81.82) - open/close 4959 (time 1753742949.41 total 60.04 last 82.43) - open/close 5793 (time 1753742959.42 total 70.04 last 83.36) - open/close 6701 (time 1753742969.43 total 80.06 last 90.68) - open/close 7531 (time 1753742979.44 total 90.06 last 82.97) - open/close 8306 (time 1753742989.44 total 100.07 last 77.45) - open/close 8966 (time 1753742999.45 total 110.07 last 65.96) - open/close 9824 (time 1753743009.45 total 120.08 last 85.77) total: 10000 open/close in 122.42 seconds: 81.68 ops/second - unlinked 0 (time 1753743016 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 836 (time 1753743097.98 total 10.00 last 83.59) - open/close 1637 (time 1753743107.99 total 20.01 last 80.01) - open/close 2492 (time 1753743118.01 total 30.03 last 85.36) - open/close 3384 (time 1753743128.02 total 40.03 last 89.15) - open/close 4271 (time 1753743138.02 total 50.04 last 88.66) - open/close 5136 (time 1753743148.03 total 60.05 last 86.42) - open/close 5973 (time 1753743158.04 total 70.06 last 83.63) - open/close 6838 (time 1753743168.05 total 80.06 last 86.44) - open/close 7679 (time 1753743178.05 total 90.07 last 84.06) - open/close 8516 (time 1753743188.06 total 100.08 last 83.64) - open/close 9373 (time 1753743198.06 total 110.08 last 85.69) total: 10000 open/close in 117.29 seconds: 85.26 ops/second - unlinked 0 (time 1753743209 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 765 (time 1753743296.96 total 10.00 last 76.49) - open/close 1582 (time 1753743306.99 total 20.03 last 81.45) - open/close 2266 (time 1753743317.00 total 30.04 last 68.37) - open/close 3039 (time 1753743327.01 total 40.05 last 77.17) - open/close 3828 (time 1753743337.03 total 50.07 last 78.76) - open/close 4722 (time 1753743347.04 total 60.08 last 89.30) - open/close 5566 (time 1753743357.04 total 70.09 last 84.37) - open/close 6512 (time 1753743367.05 total 80.09 last 94.54) - open/close 7339 (time 1753743377.06 total 90.10 last 82.64) - open/close 8198 (time 1753743387.06 total 100.10 last 85.87) - open/close 8982 (time 1753743397.06 total 110.10 last 78.38) - open/close 9695 (time 1753743407.07 total 120.11 last 71.25) total: 10000 open/close in 124.57 seconds: 80.27 ops/second - unlinked 0 (time 1753743416 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 860 (time 1753743499.51 total 10.00 last 85.97) - open/close 1754 (time 1753743509.52 total 20.01 last 89.32) - open/close 2574 (time 1753743519.52 total 30.02 last 81.96) - open/close 3405 (time 1753743529.53 total 40.02 last 83.06) - open/close 4270 (time 1753743539.54 total 50.03 last 86.42) - open/close 5151 (time 1753743549.54 total 60.03 last 88.10) - open/close 6048 (time 1753743559.55 total 70.04 last 89.63) - open/close 6935 (time 1753743569.55 total 80.05 last 88.64) - open/close 7792 (time 1753743579.56 total 90.06 last 85.62) - open/close 8605 (time 1753743589.57 total 100.06 last 81.29) - open/close 9429 (time 1753743599.58 total 110.07 last 82.30) total: 10000 open/close in 116.63 seconds: 85.74 ops/second - unlinked 0 (time 1753743609 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 854 (time 1753743691.78 total 10.02 last 85.24) - open/close 1755 (time 1753743701.79 total 20.03 last 90.01) - open/close 2656 (time 1753743711.80 total 30.04 last 90.02) - open/close 3569 (time 1753743721.80 total 40.05 last 91.23) - open/close 4450 (time 1753743731.81 total 50.05 last 88.04) - open/close 5295 (time 1753743741.81 total 60.05 last 84.50) - open/close 6195 (time 1753743751.82 total 70.06 last 89.96) - open/close 7065 (time 1753743761.82 total 80.06 last 86.97) - open/close 7909 (time 1753743771.82 total 90.06 last 84.39) - open/close 8810 (time 1753743781.82 total 100.07 last 90.05) - open/close 9724 (time 1753743791.83 total 110.07 last 91.32) total: 10000 open/close in 113.46 seconds: 88.14 ops/second - unlinked 0 (time 1753743798 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1007 (time 1753743877.06 total 10.01 last 100.65) - open/close 1921 (time 1753743887.07 total 20.02 last 91.25) - open/close 2781 (time 1753743897.08 total 30.03 last 85.95) - open/close 3631 (time 1753743907.09 total 40.03 last 84.95) - open/close 4520 (time 1753743917.10 total 50.04 last 88.82) - open/close 5434 (time 1753743927.10 total 60.04 last 91.38) - open/close 6356 (time 1753743937.10 total 70.05 last 92.17) - open/close 7277 (time 1753743947.11 total 80.05 last 92.05) - open/close 8170 (time 1753743957.11 total 90.06 last 89.26) - open/close 9026 (time 1753743967.12 total 100.06 last 85.57) - open/close 9920 (time 1753743977.13 total 110.07 last 89.31) total: 10000 open/close in 111.12 seconds: 89.99 ops/second - unlinked 0 (time 1753743982 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 872 (time 1753744067.44 total 10.01 last 87.08) - open/close 1733 (time 1753744077.44 total 20.02 last 86.09) - open/close 2633 (time 1753744087.45 total 30.02 last 89.94) - open/close 3503 (time 1753744097.45 total 40.03 last 86.97) - open/close 4386 (time 1753744107.46 total 50.03 last 88.26) - open/close 5308 (time 1753744117.46 total 60.03 last 92.16) - open/close 6203 (time 1753744127.47 total 70.04 last 89.45) - open/close 7089 (time 1753744137.49 total 80.06 last 88.45) - open/close 7977 (time 1753744147.49 total 90.06 last 88.74) - open/close 8852 (time 1753744157.50 total 100.07 last 87.46) - open/close 9666 (time 1753744167.50 total 110.07 last 81.40) total: 10000 open/close in 114.10 seconds: 87.65 ops/second - unlinked 0 (time 1753744175 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 902 (time 1753744253.55 total 10.01 last 90.12) - open/close 1798 (time 1753744263.56 total 20.01 last 89.57) - open/close 2699 (time 1753744273.57 total 30.02 last 90.02) - open/close 3575 (time 1753744283.57 total 40.03 last 87.56) - open/close 4494 (time 1753744293.57 total 50.03 last 91.87) - open/close 5369 (time 1753744303.59 total 60.04 last 87.36) - open/close 6386 (time 1753744313.60 total 70.05 last 101.64) - open/close 7422 (time 1753744323.60 total 80.06 last 103.55) - open/close 8423 (time 1753744333.62 total 90.08 last 99.89) - open/close 9432 (time 1753744343.63 total 100.09 last 100.80) total: 10000 open/close in 105.84 seconds: 94.48 ops/second - unlinked 0 (time 1753744351 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 965 (time 1753744420.83 total 10.00 last 96.49) - open/close 1974 (time 1753744430.84 total 20.01 last 100.81) - open/close 2913 (time 1753744440.85 total 30.02 last 93.80) - open/close 3879 (time 1753744450.86 total 40.03 last 96.53) - open/close 4738 (time 1753744460.87 total 50.04 last 85.83) - open/close 5625 (time 1753744470.87 total 60.04 last 88.65) - open/close 6308 (time 1753744480.88 total 70.05 last 68.24) - open/close 7013 (time 1753744490.90 total 80.07 last 70.39) - open/close 7873 (time 1753744500.90 total 90.07 last 85.97) - open/close 8794 (time 1753744510.91 total 100.08 last 92.05) - open/close 9573 (time 1753744520.91 total 110.08 last 77.89) total: 10000 open/close in 114.49 seconds: 87.34 ops/second - unlinked 0 (time 1753744528 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 846 (time 1753744610.24 total 10.01 last 84.53) - open/close 1716 (time 1753744620.24 total 20.02 last 86.93) - open/close 2588 (time 1753744630.25 total 30.02 last 87.18) - open/close 3457 (time 1753744640.26 total 40.03 last 86.76) - open/close 4347 (time 1753744650.28 total 50.05 last 88.88) - open/close 5248 (time 1753744660.28 total 60.06 last 90.03) - open/close 6129 (time 1753744670.29 total 70.06 last 88.08) - open/close 6995 (time 1753744680.29 total 80.06 last 86.58) - open/close 7792 (time 1753744690.29 total 90.07 last 79.66) - open/close 8646 (time 1753744700.29 total 100.07 last 85.39) - open/close 9499 (time 1753744710.30 total 110.07 last 85.27) total: 10000 open/close in 115.67 seconds: 86.45 ops/second - unlinked 0 (time 1753744719 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg119-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2