== conf-sanity test 69: replace an OST with the same index ========================================================== 21:27:29 (1767320849) start mds service on oleg342-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg342-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions oleg342-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg342-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 685 (time 1767320982.12 total 10.00 last 68.50) - open/close 1351 (time 1767320992.13 total 20.02 last 66.49) - open/close 2053 (time 1767321002.14 total 30.02 last 70.15) - open/close 2842 (time 1767321012.15 total 40.03 last 78.83) - open/close 3535 (time 1767321022.16 total 50.04 last 69.22) - open/close 4268 (time 1767321032.17 total 60.06 last 73.21) - open/close 5050 (time 1767321042.19 total 70.07 last 78.11) - open/close 5735 (time 1767321052.19 total 80.08 last 68.45) - open/close 6456 (time 1767321062.20 total 90.08 last 72.07) - open/close 7126 (time 1767321072.20 total 100.09 last 66.96) - open/close 7733 (time 1767321082.21 total 110.09 last 60.68) - open/close 8461 (time 1767321092.21 total 120.10 last 72.74) - open/close 9174 (time 1767321102.23 total 130.11 last 71.22) - open/close 9878 (time 1767321112.24 total 140.12 last 70.34) total: 10000 open/close in 141.67 seconds: 70.59 ops/second - unlinked 0 (time 1767321118 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 792 (time 1767321210.46 total 10.00 last 79.18) - open/close 1471 (time 1767321220.46 total 20.01 last 67.87) - open/close 2184 (time 1767321230.47 total 30.01 last 71.26) - open/close 2904 (time 1767321240.48 total 40.02 last 71.95) - open/close 3579 (time 1767321250.48 total 50.03 last 67.46) - open/close 4337 (time 1767321260.48 total 60.03 last 75.78) - open/close 5150 (time 1767321270.49 total 70.03 last 81.27) - open/close 5908 (time 1767321280.49 total 80.03 last 75.80) - open/close 6684 (time 1767321290.50 total 90.04 last 77.53) - open/close 7451 (time 1767321300.50 total 100.04 last 76.68) - open/close 8158 (time 1767321310.50 total 110.04 last 70.70) - open/close 8964 (time 1767321320.51 total 120.05 last 80.56) - open/close 9712 (time 1767321330.53 total 130.07 last 74.64) total: 10000 open/close in 134.11 seconds: 74.57 ops/second - unlinked 0 (time 1767321338 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 771 (time 1767321427.73 total 10.00 last 77.10) - open/close 1433 (time 1767321437.74 total 20.00 last 66.19) - open/close 2135 (time 1767321447.75 total 30.02 last 70.10) - open/close 2907 (time 1767321457.75 total 40.02 last 77.19) - open/close 3635 (time 1767321467.76 total 50.02 last 72.75) - open/close 4390 (time 1767321477.77 total 60.04 last 75.38) - open/close 5230 (time 1767321487.78 total 70.05 last 83.92) - open/close 5973 (time 1767321497.79 total 80.05 last 74.27) - open/close 6689 (time 1767321507.80 total 90.06 last 71.53) - open/close 7416 (time 1767321517.80 total 100.07 last 72.65) - open/close 8112 (time 1767321527.80 total 110.07 last 69.59) - open/close 8837 (time 1767321537.81 total 120.08 last 72.48) - open/close 9523 (time 1767321547.82 total 130.09 last 68.51) total: 10000 open/close in 136.08 seconds: 73.48 ops/second - unlinked 0 (time 1767321558 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 703 (time 1767321647.20 total 10.00 last 70.30) - open/close 1416 (time 1767321657.21 total 20.01 last 71.21) - open/close 2270 (time 1767321667.23 total 30.02 last 85.30) - open/close 3024 (time 1767321677.23 total 40.03 last 75.39) - open/close 3732 (time 1767321687.24 total 50.04 last 70.69) - open/close 4430 (time 1767321697.25 total 60.04 last 69.78) - open/close 5165 (time 1767321707.26 total 70.06 last 73.41) - open/close 5878 (time 1767321717.26 total 80.06 last 71.27) - open/close 6695 (time 1767321727.28 total 90.08 last 81.54) - open/close 7511 (time 1767321737.28 total 100.08 last 81.58) - open/close 8279 (time 1767321747.29 total 110.09 last 76.72) - open/close 9024 (time 1767321757.30 total 120.10 last 74.43) - open/close 9612 (time 1767321767.31 total 130.11 last 58.74) total: 10000 open/close in 135.39 seconds: 73.86 ops/second - unlinked 0 (time 1767321777 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 735 (time 1767321867.94 total 10.00 last 73.49) - open/close 1502 (time 1767321877.97 total 20.02 last 76.53) - open/close 2266 (time 1767321887.98 total 30.04 last 76.28) - open/close 3071 (time 1767321897.99 total 40.04 last 80.47) - open/close 3914 (time 1767321908.00 total 50.05 last 84.22) - open/close 4786 (time 1767321918.00 total 60.06 last 87.13) - open/close 5493 (time 1767321928.01 total 70.06 last 70.68) - open/close 6247 (time 1767321938.02 total 80.08 last 75.31) - open/close 6914 (time 1767321948.02 total 90.08 last 66.68) - open/close 7630 (time 1767321958.03 total 100.09 last 71.54) - open/close 8320 (time 1767321968.04 total 110.10 last 68.91) - open/close 9031 (time 1767321978.05 total 120.11 last 71.01) - open/close 9773 (time 1767321988.06 total 130.11 last 74.20) total: 10000 open/close in 133.08 seconds: 75.14 ops/second - unlinked 0 (time 1767321997 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 777 (time 1767322097.42 total 10.00 last 77.69) - open/close 1492 (time 1767322107.42 total 20.00 last 71.47) - open/close 2226 (time 1767322117.42 total 30.01 last 73.37) - open/close 2957 (time 1767322127.43 total 40.02 last 73.04) - open/close 3694 (time 1767322137.44 total 50.02 last 73.67) - open/close 4450 (time 1767322147.47 total 60.06 last 75.31) - open/close 5182 (time 1767322157.47 total 70.06 last 73.20) - open/close 5956 (time 1767322167.47 total 80.06 last 77.40) - open/close 6657 (time 1767322177.48 total 90.06 last 70.08) - open/close 7409 (time 1767322187.48 total 100.07 last 75.16) - open/close 8180 (time 1767322197.48 total 110.07 last 77.09) - open/close 8910 (time 1767322207.49 total 120.08 last 72.93) - open/close 9608 (time 1767322217.50 total 130.09 last 69.74) total: 10000 open/close in 135.08 seconds: 74.03 ops/second - unlinked 0 (time 1767322227 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 742 (time 1767322319.99 total 10.01 last 74.11) - open/close 1489 (time 1767322330.02 total 20.04 last 74.50) - open/close 2163 (time 1767322340.03 total 30.05 last 67.35) - open/close 2819 (time 1767322350.03 total 40.05 last 65.57) - open/close 3580 (time 1767322360.05 total 50.07 last 75.98) - open/close 4287 (time 1767322370.06 total 60.08 last 70.64) - open/close 5030 (time 1767322380.06 total 70.08 last 74.25) - open/close 5691 (time 1767322390.06 total 80.08 last 66.09) - open/close 6451 (time 1767322400.07 total 90.09 last 75.94) - open/close 7184 (time 1767322410.08 total 100.10 last 73.21) - open/close 7950 (time 1767322420.09 total 110.11 last 76.55) - open/close 8747 (time 1767322430.10 total 120.12 last 79.61) - open/close 9513 (time 1767322440.11 total 130.13 last 76.53) total: 10000 open/close in 137.33 seconds: 72.82 ops/second - unlinked 0 (time 1767322451 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 792 (time 1767322544.99 total 10.01 last 79.16) - open/close 1637 (time 1767322555.00 total 20.02 last 84.41) - open/close 2387 (time 1767322565.01 total 30.03 last 74.91) - open/close 3065 (time 1767322575.03 total 40.04 last 67.70) - open/close 3753 (time 1767322585.04 total 50.05 last 68.74) - open/close 4584 (time 1767322595.05 total 60.07 last 82.98) - open/close 5368 (time 1767322605.06 total 70.07 last 78.38) - open/close 6178 (time 1767322615.06 total 80.08 last 80.92) - open/close 6903 (time 1767322625.08 total 90.09 last 72.41) - open/close 7644 (time 1767322635.09 total 100.10 last 74.03) - open/close 8344 (time 1767322645.10 total 110.11 last 69.94) - open/close 8907 (time 1767322655.10 total 120.11 last 56.30) - open/close 9730 (time 1767322665.11 total 130.12 last 82.22) total: 10000 open/close in 134.33 seconds: 74.44 ops/second - unlinked 0 (time 1767322673 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 871 (time 1767322752.87 total 10.01 last 87.06) - open/close 1711 (time 1767322762.88 total 20.01 last 83.95) - open/close 2593 (time 1767322772.88 total 30.01 last 88.18) - open/close 3478 (time 1767322782.88 total 40.02 last 88.48) - open/close 4459 (time 1767322792.89 total 50.02 last 98.02) - open/close 5444 (time 1767322802.89 total 60.03 last 98.47) - open/close 6392 (time 1767322812.90 total 70.04 last 94.70) - open/close 7266 (time 1767322822.91 total 80.05 last 87.34) - open/close 8159 (time 1767322832.91 total 90.05 last 89.29) - open/close 8966 (time 1767322842.92 total 100.06 last 80.62) - open/close 9774 (time 1767322852.93 total 110.06 last 80.74) total: 10000 open/close in 112.35 seconds: 89.01 ops/second - unlinked 0 (time 1767322858 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 904 (time 1767322928.52 total 10.01 last 90.34) - open/close 1832 (time 1767322938.52 total 20.01 last 92.76) - open/close 2760 (time 1767322948.53 total 30.02 last 92.75) - open/close 3609 (time 1767322958.53 total 40.02 last 84.84) - open/close 4514 (time 1767322968.54 total 50.03 last 90.44) - open/close 5640 (time 1767322978.54 total 60.03 last 112.56) - open/close 6661 (time 1767322988.55 total 70.04 last 102.03) - open/close 7670 (time 1767322998.55 total 80.04 last 100.87) - open/close 8693 (time 1767323008.56 total 90.05 last 102.17) - open/close 9654 (time 1767323018.57 total 100.06 last 96.06) total: 10000 open/close in 103.65 seconds: 96.48 ops/second - unlinked 0 (time 1767323025 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre On OST0, 17829 used inodes rc=0 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.142@tcp autodegrade=on sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.142@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.142@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg342-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local oleg342-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg342-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2