== conf-sanity test 69: replace an OST with the same index ========================================================== 05:30:34 (1765103434) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 802 (time 1765103501.24 total 10.00 last 80.17) - open/close 1568 (time 1765103511.26 total 20.02 last 76.44) - open/close 2325 (time 1765103521.27 total 30.03 last 75.64) - open/close 3091 (time 1765103531.28 total 40.05 last 76.49) - open/close 3882 (time 1765103541.28 total 50.05 last 79.10) - open/close 4708 (time 1765103551.29 total 60.06 last 82.51) - open/close 5441 (time 1765103561.29 total 70.06 last 73.29) - open/close 6153 (time 1765103571.30 total 80.06 last 71.18) - open/close 6894 (time 1765103581.30 total 90.07 last 74.07) - open/close 7690 (time 1765103591.31 total 100.08 last 79.49) - open/close 8370 (time 1765103601.32 total 110.09 last 67.96) - open/close 9124 (time 1765103611.34 total 120.11 last 75.26) - open/close 9831 (time 1765103621.35 total 130.11 last 70.66) total: 10000 open/close in 132.40 seconds: 75.53 ops/second - unlinked 0 (time 1765103626 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 902 (time 1765103708.94 total 10.00 last 90.18) - open/close 1735 (time 1765103718.94 total 20.00 last 83.28) - open/close 2728 (time 1765103728.95 total 30.01 last 99.24) - open/close 3812 (time 1765103738.96 total 40.02 last 108.28) - open/close 4765 (time 1765103748.96 total 50.02 last 95.29) - open/close 5711 (time 1765103758.97 total 60.03 last 94.51) - open/close 6591 (time 1765103768.97 total 70.04 last 87.97) - open/close 7549 (time 1765103778.97 total 80.04 last 95.79) - open/close 8681 (time 1765103788.98 total 90.04 last 113.11) - open/close 9492 (time 1765103798.99 total 100.05 last 81.02) total: 10000 open/close in 106.88 seconds: 93.57 ops/second - unlinked 0 (time 1765103809 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1248 (time 1765103884.52 total 10.01 last 124.70) - open/close 2185 (time 1765103894.52 total 20.01 last 93.68) - open/close 3096 (time 1765103904.53 total 30.02 last 91.06) - open/close 4007 (time 1765103914.53 total 40.02 last 91.06) - open/close 4909 (time 1765103924.54 total 50.02 last 90.16) - open/close 5916 (time 1765103934.55 total 60.04 last 100.59) - open/close 6956 (time 1765103944.55 total 70.04 last 103.95) - open/close 8192 (time 1765103954.56 total 80.05 last 123.51) - open/close 9261 (time 1765103964.56 total 90.05 last 106.84) total: 10000 open/close in 96.53 seconds: 103.60 ops/second - unlinked 0 (time 1765103972 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 841 (time 1765104044.23 total 10.01 last 84.05) - open/close 1819 (time 1765104054.25 total 20.02 last 97.64) - open/close 2829 (time 1765104064.25 total 30.03 last 100.95) - open/close 3744 (time 1765104074.26 total 40.03 last 91.46) - open/close 4604 (time 1765104084.27 total 50.04 last 85.90) - open/close 5423 (time 1765104094.27 total 60.04 last 81.90) - open/close 6371 (time 1765104104.28 total 70.05 last 94.74) - open/close 7299 (time 1765104114.29 total 80.06 last 92.70) - open/close 8308 (time 1765104124.29 total 90.06 last 100.90) - open/close 9284 (time 1765104134.29 total 100.06 last 97.58) total: 10000 open/close in 107.40 seconds: 93.11 ops/second - unlinked 0 (time 1765104143 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1197 (time 1765104213.56 total 10.00 last 119.69) - open/close 2169 (time 1765104223.57 total 20.01 last 97.10) - open/close 3199 (time 1765104233.58 total 30.02 last 102.94) - open/close 4331 (time 1765104243.59 total 40.02 last 113.13) - open/close 5356 (time 1765104253.59 total 50.02 last 102.50) - open/close 6490 (time 1765104263.59 total 60.03 last 113.37) - open/close 7437 (time 1765104273.60 total 70.04 last 94.60) - open/close 8408 (time 1765104283.61 total 80.05 last 96.99) - open/close 9288 (time 1765104293.61 total 90.05 last 87.97) total: 10000 open/close in 97.86 seconds: 102.18 ops/second - unlinked 0 (time 1765104304 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 998 (time 1765104368.21 total 10.01 last 99.73) - open/close 1935 (time 1765104378.21 total 20.01 last 93.65) - open/close 2878 (time 1765104388.22 total 30.02 last 94.22) - open/close 3897 (time 1765104398.23 total 40.03 last 101.84) - open/close 4936 (time 1765104408.24 total 50.04 last 103.80) - open/close 5864 (time 1765104418.24 total 60.04 last 92.72) - open/close 6870 (time 1765104428.25 total 70.05 last 100.59) - open/close 7972 (time 1765104438.25 total 80.05 last 110.13) - open/close 9156 (time 1765104448.25 total 90.05 last 118.39) total: 10000 open/close in 99.01 seconds: 101.00 ops/second - unlinked 0 (time 1765104459 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1390 (time 1765104519.57 total 10.01 last 138.93) - open/close 2771 (time 1765104529.57 total 20.01 last 138.06) - open/close 4022 (time 1765104539.58 total 30.01 last 125.02) - open/close 5307 (time 1765104549.60 total 40.03 last 128.28) - open/close 6720 (time 1765104559.60 total 50.04 last 141.20) - open/close 7991 (time 1765104569.60 total 60.04 last 127.08) - open/close 9236 (time 1765104579.61 total 70.05 last 124.42) total: 10000 open/close in 75.81 seconds: 131.91 ops/second - unlinked 0 (time 1765104587 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 971 (time 1765104652.96 total 10.00 last 97.08) - open/close 1965 (time 1765104662.97 total 20.01 last 99.36) - open/close 2990 (time 1765104672.97 total 30.01 last 102.48) - open/close 3969 (time 1765104682.98 total 40.02 last 97.81) - open/close 4937 (time 1765104692.98 total 50.02 last 96.79) - open/close 5860 (time 1765104702.99 total 60.03 last 92.18) - open/close 6807 (time 1765104713.00 total 70.03 last 94.68) - open/close 7670 (time 1765104723.00 total 80.04 last 86.27) - open/close 8558 (time 1765104733.01 total 90.05 last 88.68) - open/close 9618 (time 1765104743.02 total 100.06 last 105.88) total: 10000 open/close in 103.81 seconds: 96.33 ops/second - unlinked 0 (time 1765104749 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1038 (time 1765104809.77 total 10.00 last 103.78) - open/close 1973 (time 1765104819.78 total 20.01 last 93.42) - open/close 2786 (time 1765104829.79 total 30.02 last 81.25) - open/close 3740 (time 1765104839.80 total 40.02 last 95.34) - open/close 4613 (time 1765104849.80 total 50.03 last 87.23) - open/close 5461 (time 1765104859.81 total 60.04 last 84.74) - open/close 6356 (time 1765104869.81 total 70.04 last 89.49) - open/close 7391 (time 1765104879.82 total 80.04 last 103.46) - open/close 8225 (time 1765104889.82 total 90.05 last 83.33) - open/close 9119 (time 1765104899.83 total 100.06 last 89.34) total: 10000 open/close in 109.28 seconds: 91.51 ops/second - unlinked 0 (time 1765104911 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 937 (time 1765104982.09 total 10.00 last 93.66) - open/close 1925 (time 1765104992.09 total 20.01 last 98.78) - open/close 2874 (time 1765105002.12 total 30.03 last 94.67) - open/close 3875 (time 1765105012.12 total 40.03 last 100.06) - open/close 4916 (time 1765105022.13 total 50.05 last 103.98) - open/close 6090 (time 1765105032.14 total 60.05 last 117.37) - open/close 7123 (time 1765105042.14 total 70.05 last 103.26) - open/close 8276 (time 1765105052.15 total 80.06 last 115.25) - open/close 9389 (time 1765105062.15 total 90.06 last 111.29) total: 10000 open/close in 94.90 seconds: 105.38 ops/second - unlinked 0 (time 1765105069 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre On OST0, 5565 used inodes rc=0 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.144@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg444-server: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg444-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2