== conf-sanity test 69: replace an OST with the same index ========================================================== 14:29:12 (1773685752) start mds service on oleg460-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg460-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99872. rc=0 - open/close 1364 (time 1773685814.38 total 10.01 last 136.32) - open/close 2530 (time 1773685824.38 total 20.01 last 116.57) - open/close 3777 (time 1773685834.39 total 30.02 last 124.59) - open/close 5292 (time 1773685844.39 total 40.02 last 151.49) - open/close 6586 (time 1773685854.39 total 50.02 last 129.34) - open/close 7908 (time 1773685864.40 total 60.03 last 132.08) - open/close 9141 (time 1773685874.40 total 70.03 last 123.26) total: 10000 open/close in 76.92 seconds: 130.01 ops/second - unlinked 0 (time 1773685883 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1195 (time 1773685942.21 total 10.00 last 119.48) - open/close 2402 (time 1773685952.21 total 20.00 last 120.69) - open/close 3762 (time 1773685962.21 total 30.00 last 135.98) - open/close 4970 (time 1773685972.21 total 40.01 last 120.77) - open/close 6296 (time 1773685982.21 total 50.01 last 132.59) - open/close 7593 (time 1773685992.21 total 60.01 last 129.68) - open/close 8794 (time 1773686002.22 total 70.02 last 119.97) - open/close 9834 (time 1773686012.23 total 80.02 last 103.95) total: 10000 open/close in 81.32 seconds: 122.96 ops/second - unlinked 0 (time 1773686015 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1187 (time 1773686072.81 total 10.01 last 118.63) - open/close 2647 (time 1773686082.81 total 20.01 last 145.92) - open/close 3980 (time 1773686092.82 total 30.01 last 133.25) - open/close 5144 (time 1773686102.82 total 40.02 last 116.37) - open/close 6375 (time 1773686112.82 total 50.02 last 123.06) - open/close 7734 (time 1773686122.82 total 60.02 last 135.86) - open/close 9096 (time 1773686132.83 total 70.03 last 136.14) total: 10000 open/close in 78.08 seconds: 128.07 ops/second - unlinked 0 (time 1773686142 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1321 (time 1773686193.91 total 10.00 last 132.09) - open/close 3085 (time 1773686203.91 total 20.00 last 176.39) - open/close 5081 (time 1773686213.91 total 30.00 last 199.54) - open/close 6866 (time 1773686223.93 total 40.02 last 178.29) - open/close 8103 (time 1773686233.93 total 50.02 last 123.62) - open/close 9433 (time 1773686243.93 total 60.02 last 133.00) total: 10000 open/close in 64.69 seconds: 154.57 ops/second - unlinked 0 (time 1773686250 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1771 (time 1773686301.33 total 10.00 last 177.06) - open/close 3248 (time 1773686311.34 total 20.01 last 147.62) - open/close 4690 (time 1773686321.34 total 30.01 last 144.18) - open/close 6675 (time 1773686331.34 total 40.01 last 198.47) - open/close 8627 (time 1773686341.34 total 50.01 last 195.14) total: 10000 open/close in 58.24 seconds: 171.71 ops/second - unlinked 0 (time 1773686351 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1537 (time 1773686399.25 total 10.00 last 153.66) - open/close 3378 (time 1773686409.26 total 20.01 last 184.05) - open/close 5462 (time 1773686419.26 total 30.01 last 208.32) - open/close 7046 (time 1773686429.26 total 40.01 last 158.39) - open/close 8549 (time 1773686439.27 total 50.01 last 150.23) total: 10000 open/close in 58.84 seconds: 169.94 ops/second - unlinked 0 (time 1773686449 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1533 (time 1773686497.58 total 10.01 last 153.15) - open/close 3132 (time 1773686507.59 total 20.01 last 159.82) - open/close 4772 (time 1773686517.59 total 30.02 last 163.99) - open/close 6799 (time 1773686527.59 total 40.02 last 202.67) - open/close 8605 (time 1773686537.60 total 50.02 last 180.46) total: 10000 open/close in 57.58 seconds: 173.67 ops/second - unlinked 0 (time 1773686546 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2211 (time 1773686586.70 total 10.00 last 221.02) - open/close 4462 (time 1773686596.70 total 20.01 last 225.04) - open/close 5960 (time 1773686606.71 total 30.01 last 149.74) - open/close 7514 (time 1773686616.71 total 40.01 last 155.33) total: 10000 open/close in 49.72 seconds: 201.14 ops/second - unlinked 0 (time 1773686628 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 2726 (time 1773686668.41 total 10.00 last 272.59) - open/close 5648 (time 1773686678.42 total 20.01 last 291.99) - open/close 8042 (time 1773686688.42 total 30.01 last 239.36) total: 10000 open/close in 38.77 seconds: 257.91 ops/second - unlinked 0 (time 1773686698 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 2337 (time 1773686741.95 total 10.00 last 233.68) - open/close 4712 (time 1773686751.95 total 20.00 last 237.49) - open/close 7249 (time 1773686761.96 total 30.01 last 253.54) - open/close 9747 (time 1773686771.96 total 40.01 last 249.77) total: 10000 open/close in 40.74 seconds: 245.43 ops/second - unlinked 0 (time 1773686774 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre On OST0, 2807 used inodes rc=0 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.160@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.160@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.204.160@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2