== conf-sanity test 69: replace an OST with the same index ========================================================== 15:27:21 (1768249641) start mds service on oleg117-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 625 (time 1768249729.57 total 10.02 last 62.40) - open/close 1277 (time 1768249739.60 total 20.04 last 65.03) - open/close 1952 (time 1768249749.60 total 30.04 last 67.50) - open/close 2565 (time 1768249759.61 total 40.05 last 61.24) - open/close 3065 (time 1768249769.62 total 50.07 last 49.93) - open/close 3694 (time 1768249779.63 total 60.08 last 62.83) - open/close 4275 (time 1768249789.66 total 70.11 last 57.94) - open/close 4796 (time 1768249799.69 total 80.13 last 51.97) - open/close 5355 (time 1768249809.71 total 90.15 last 55.78) - open/close 5857 (time 1768249819.71 total 100.15 last 50.20) - open/close 6444 (time 1768249829.71 total 110.15 last 58.68) - open/close 7101 (time 1768249839.72 total 120.17 last 65.63) - open/close 7717 (time 1768249849.74 total 130.18 last 61.50) - open/close 8340 (time 1768249859.75 total 140.19 last 62.23) - open/close 8964 (time 1768249869.76 total 150.20 last 62.35) - open/close 9568 (time 1768249879.76 total 160.20 last 60.38) total: 10000 open/close in 167.55 seconds: 59.68 ops/second - unlinked 0 (time 1768249892 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 559 (time 1768250004.29 total 10.00 last 55.88) - open/close 1016 (time 1768250014.30 total 20.02 last 45.63) - open/close 1726 (time 1768250024.31 total 30.03 last 70.93) - open/close 2313 (time 1768250034.32 total 40.04 last 58.64) - open/close 3067 (time 1768250044.33 total 50.04 last 75.35) - open/close 3798 (time 1768250054.34 total 60.05 last 73.04) - open/close 4418 (time 1768250064.35 total 70.07 last 61.91) - open/close 4853 (time 1768250074.36 total 80.08 last 43.46) - open/close 5346 (time 1768250084.37 total 90.08 last 49.26) - open/close 5746 (time 1768250094.38 total 100.10 last 39.95) - open/close 6168 (time 1768250104.39 total 110.10 last 42.18) - open/close 6585 (time 1768250114.40 total 120.11 last 41.67) - open/close 7127 (time 1768250124.40 total 130.12 last 54.17) - open/close 7733 (time 1768250134.42 total 140.13 last 60.49) - open/close 8252 (time 1768250144.42 total 150.14 last 51.89) - open/close 8747 (time 1768250154.44 total 160.15 last 49.44) - open/close 9397 (time 1768250164.45 total 170.17 last 64.88) - open/close 9944 (time 1768250174.46 total 180.17 last 54.67) total: 10000 open/close in 181.20 seconds: 55.19 ops/second - unlinked 0 (time 1768250180 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 579 (time 1768250299.49 total 10.04 last 57.70) - open/close 1187 (time 1768250309.49 total 20.04 last 60.77) - open/close 1902 (time 1768250319.50 total 30.05 last 71.43) - open/close 2549 (time 1768250329.51 total 40.06 last 64.65) - open/close 3141 (time 1768250339.52 total 50.07 last 59.14) - open/close 3762 (time 1768250349.52 total 60.07 last 62.09) - open/close 4368 (time 1768250359.54 total 70.08 last 60.51) - open/close 5007 (time 1768250369.55 total 80.10 last 63.82) - open/close 5700 (time 1768250379.56 total 90.11 last 69.20) - open/close 6424 (time 1768250389.57 total 100.11 last 72.38) - open/close 7022 (time 1768250399.57 total 110.12 last 59.77) - open/close 7596 (time 1768250409.57 total 120.12 last 57.37) - open/close 8149 (time 1768250419.58 total 130.12 last 55.29) - open/close 8737 (time 1768250429.59 total 140.14 last 58.70) - open/close 9139 (time 1768250439.61 total 150.16 last 40.14) - open/close 9607 (time 1768250449.61 total 160.16 last 46.80) total: 10000 open/close in 167.03 seconds: 59.87 ops/second - unlinked 0 (time 1768250460 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 785 (time 1768250574.40 total 10.01 last 78.41) - open/close 1434 (time 1768250584.41 total 20.01 last 64.89) - open/close 2032 (time 1768250594.41 total 30.02 last 59.78) - open/close 2542 (time 1768250604.43 total 40.03 last 50.90) - open/close 3211 (time 1768250614.43 total 50.04 last 66.89) - open/close 3827 (time 1768250624.43 total 60.04 last 61.58) - open/close 4438 (time 1768250634.46 total 70.07 last 60.94) - open/close 5065 (time 1768250644.46 total 80.07 last 62.67) - open/close 5803 (time 1768250654.47 total 90.07 last 73.77) - open/close 6413 (time 1768250664.47 total 100.08 last 60.96) - open/close 7094 (time 1768250674.48 total 110.09 last 68.03) - open/close 7772 (time 1768250684.49 total 120.10 last 67.77) - open/close 8412 (time 1768250694.50 total 130.10 last 63.95) - open/close 9057 (time 1768250704.50 total 140.11 last 64.45) - open/close 9630 (time 1768250714.52 total 150.13 last 57.21) total: 10000 open/close in 155.19 seconds: 64.44 ops/second - unlinked 0 (time 1768250724 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 616 (time 1768250835.60 total 10.01 last 61.53) - open/close 1272 (time 1768250845.62 total 20.03 last 65.49) - open/close 2012 (time 1768250855.63 total 30.04 last 73.91) - open/close 2758 (time 1768250865.65 total 40.05 last 74.52) - open/close 3336 (time 1768250875.66 total 50.07 last 57.72) - open/close 3706 (time 1768250885.68 total 60.09 last 36.93) - open/close 4238 (time 1768250895.70 total 70.10 last 53.10) - open/close 4952 (time 1768250905.71 total 80.12 last 71.30) - open/close 5635 (time 1768250915.71 total 90.12 last 68.29) - open/close 6306 (time 1768250925.71 total 100.12 last 67.10) - open/close 7161 (time 1768250935.72 total 110.12 last 85.48) - open/close 7788 (time 1768250945.73 total 120.14 last 62.59) - open/close 8538 (time 1768250955.74 total 130.14 last 74.98) - open/close 9279 (time 1768250965.74 total 140.14 last 74.09) - open/close 9918 (time 1768250975.75 total 150.16 last 63.82) total: 10000 open/close in 151.57 seconds: 65.98 ops/second - unlinked 0 (time 1768250982 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 789 (time 1768251080.49 total 10.00 last 78.86) - open/close 1497 (time 1768251090.49 total 20.01 last 70.80) - open/close 2225 (time 1768251100.50 total 30.02 last 72.70) - open/close 2935 (time 1768251110.53 total 40.04 last 70.82) - open/close 3619 (time 1768251120.53 total 50.05 last 68.38) - open/close 4168 (time 1768251130.53 total 60.05 last 54.89) - open/close 4742 (time 1768251140.53 total 70.05 last 57.37) - open/close 5480 (time 1768251150.54 total 80.06 last 73.78) - open/close 6121 (time 1768251160.54 total 90.06 last 64.09) - open/close 6767 (time 1768251170.54 total 100.06 last 64.57) - open/close 7246 (time 1768251180.56 total 110.08 last 47.80) - open/close 7837 (time 1768251190.57 total 120.09 last 59.04) - open/close 8405 (time 1768251200.59 total 130.11 last 56.68) - open/close 9017 (time 1768251210.60 total 140.12 last 61.17) - open/close 9650 (time 1768251220.61 total 150.12 last 63.25) total: 10000 open/close in 155.47 seconds: 64.32 ops/second - unlinked 0 (time 1768251231 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 532 (time 1768251342.93 total 10.01 last 53.16) - open/close 1149 (time 1768251352.94 total 20.02 last 61.62) - open/close 1904 (time 1768251362.95 total 30.03 last 75.43) - open/close 2607 (time 1768251372.96 total 40.04 last 70.24) - open/close 3338 (time 1768251382.97 total 50.05 last 73.01) - open/close 4100 (time 1768251392.99 total 60.06 last 76.11) - open/close 4749 (time 1768251403.00 total 70.08 last 64.82) - open/close 5457 (time 1768251413.03 total 80.10 last 70.60) - open/close 6131 (time 1768251423.03 total 90.11 last 67.39) - open/close 6844 (time 1768251433.07 total 100.15 last 70.99) - open/close 7484 (time 1768251443.08 total 110.16 last 63.93) - open/close 8211 (time 1768251453.09 total 120.17 last 72.62) - open/close 8904 (time 1768251463.10 total 130.18 last 69.25) - open/close 9658 (time 1768251473.10 total 140.18 last 75.39) total: 10000 open/close in 144.88 seconds: 69.02 ops/second - unlinked 0 (time 1768251483 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 703 (time 1768251583.55 total 10.00 last 70.30) - open/close 1469 (time 1768251593.56 total 20.02 last 76.49) - open/close 2071 (time 1768251603.57 total 30.02 last 60.19) - open/close 2734 (time 1768251613.57 total 40.02 last 66.29) - open/close 3467 (time 1768251623.57 total 50.02 last 73.30) - open/close 4172 (time 1768251633.57 total 60.02 last 70.50) - open/close 4816 (time 1768251643.58 total 70.03 last 64.31) - open/close 5647 (time 1768251653.60 total 80.05 last 82.92) - open/close 6242 (time 1768251663.62 total 90.07 last 59.42) - open/close 6865 (time 1768251673.62 total 100.08 last 62.25) - open/close 7502 (time 1768251683.64 total 110.09 last 63.62) - open/close 8131 (time 1768251693.66 total 120.11 last 62.74) - open/close 8800 (time 1768251703.68 total 130.13 last 66.81) - open/close 9501 (time 1768251713.68 total 140.13 last 70.09) total: 10000 open/close in 147.52 seconds: 67.79 ops/second - unlinked 0 (time 1768251725 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 760 (time 1768251828.63 total 10.00 last 75.97) - open/close 1440 (time 1768251838.63 total 20.01 last 67.96) - open/close 2169 (time 1768251848.64 total 30.01 last 72.87) - open/close 2803 (time 1768251858.65 total 40.02 last 63.34) - open/close 3491 (time 1768251868.65 total 50.02 last 68.79) - open/close 4181 (time 1768251878.65 total 60.03 last 68.96) - open/close 4879 (time 1768251888.68 total 70.06 last 69.63) - open/close 5574 (time 1768251898.68 total 80.06 last 69.48) - open/close 6311 (time 1768251908.69 total 90.07 last 73.64) - open/close 7003 (time 1768251918.70 total 100.08 last 69.14) - open/close 7754 (time 1768251928.72 total 110.09 last 74.96) - open/close 8517 (time 1768251938.72 total 120.10 last 76.27) - open/close 9167 (time 1768251948.73 total 130.11 last 64.93) - open/close 9858 (time 1768251958.73 total 140.11 last 69.07) total: 10000 open/close in 142.00 seconds: 70.42 ops/second - unlinked 0 (time 1768251965 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 660 (time 1768252060.64 total 10.01 last 65.95) - open/close 1378 (time 1768252070.64 total 20.01 last 71.75) - open/close 2156 (time 1768252080.65 total 30.02 last 77.75) - open/close 2843 (time 1768252090.66 total 40.03 last 68.64) - open/close 3433 (time 1768252100.66 total 50.03 last 58.97) - open/close 4134 (time 1768252110.67 total 60.04 last 70.05) - open/close 4899 (time 1768252120.67 total 70.05 last 76.47) - open/close 5618 (time 1768252130.68 total 80.05 last 71.87) - open/close 6402 (time 1768252140.69 total 90.06 last 78.31) - open/close 7008 (time 1768252150.69 total 100.07 last 60.57) - open/close 7523 (time 1768252160.70 total 110.07 last 51.48) - open/close 8181 (time 1768252170.71 total 120.08 last 65.74) - open/close 8880 (time 1768252180.72 total 130.09 last 69.79) - open/close 9442 (time 1768252190.73 total 140.10 last 56.15) - open/close 9879 (time 1768252200.74 total 150.11 last 43.67) total: 10000 open/close in 152.40 seconds: 65.62 ops/second - unlinked 0 (time 1768252209 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.117@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.117@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.201.117@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg117-server: oleg117-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg117-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg117-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2