== conf-sanity test 69: replace an OST with the same index ========================================================== 07:16:17 (1764936977) start mds service on oleg259-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99904. rc=0 - open/close 740 (time 1764937057.28 total 10.01 last 73.94) - open/close 1381 (time 1764937067.28 total 20.01 last 64.10) - open/close 2065 (time 1764937077.28 total 30.01 last 68.35) - open/close 2824 (time 1764937087.29 total 40.02 last 75.84) - open/close 3568 (time 1764937097.29 total 50.03 last 74.38) - open/close 4271 (time 1764937107.31 total 60.04 last 70.17) - open/close 4927 (time 1764937117.32 total 70.05 last 65.54) - open/close 5581 (time 1764937127.32 total 80.06 last 65.39) - open/close 6250 (time 1764937137.33 total 90.06 last 66.86) - open/close 6964 (time 1764937147.34 total 100.07 last 71.32) - open/close 7654 (time 1764937157.36 total 110.09 last 68.90) - open/close 8348 (time 1764937167.36 total 120.09 last 69.39) - open/close 8959 (time 1764937177.37 total 130.10 last 61.03) - open/close 9544 (time 1764937187.39 total 140.12 last 58.39) total: 10000 open/close in 147.88 seconds: 67.62 ops/second - unlinked 0 (time 1764937199 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 616 (time 1764937300.53 total 10.01 last 61.51) - open/close 1263 (time 1764937310.53 total 20.02 last 64.69) - open/close 1919 (time 1764937320.53 total 30.02 last 65.55) - open/close 2583 (time 1764937330.54 total 40.03 last 66.37) - open/close 3282 (time 1764937340.54 total 50.03 last 69.87) - open/close 3936 (time 1764937350.56 total 60.05 last 65.31) - open/close 4577 (time 1764937360.56 total 70.05 last 64.09) - open/close 5191 (time 1764937370.57 total 80.06 last 61.34) - open/close 5822 (time 1764937380.58 total 90.07 last 63.01) - open/close 6400 (time 1764937390.59 total 100.08 last 57.76) - open/close 7045 (time 1764937400.60 total 110.08 last 64.46) - open/close 7672 (time 1764937410.61 total 120.10 last 62.63) - open/close 8294 (time 1764937420.62 total 130.10 last 62.15) - open/close 8884 (time 1764937430.62 total 140.11 last 58.98) - open/close 9499 (time 1764937440.63 total 150.12 last 61.41) total: 10000 open/close in 158.22 seconds: 63.20 ops/second - unlinked 0 (time 1764937451 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 665 (time 1764937553.68 total 10.02 last 66.40) - open/close 1306 (time 1764937563.69 total 20.03 last 64.04) - open/close 1951 (time 1764937573.69 total 30.03 last 64.49) - open/close 2663 (time 1764937583.70 total 40.04 last 71.15) - open/close 3380 (time 1764937593.72 total 50.06 last 71.54) - open/close 4042 (time 1764937603.72 total 60.06 last 66.19) - open/close 4709 (time 1764937613.74 total 70.08 last 66.59) - open/close 5366 (time 1764937623.75 total 80.09 last 65.60) - open/close 5999 (time 1764937633.76 total 90.09 last 63.28) - open/close 6672 (time 1764937643.76 total 100.10 last 67.27) - open/close 7313 (time 1764937653.78 total 110.11 last 64.00) - open/close 7899 (time 1764937663.78 total 120.12 last 58.56) - open/close 8530 (time 1764937673.79 total 130.13 last 63.03) - open/close 9190 (time 1764937683.80 total 140.13 last 65.99) - open/close 9856 (time 1764937693.81 total 150.14 last 66.53) total: 10000 open/close in 152.25 seconds: 65.68 ops/second - unlinked 0 (time 1764937699 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 688 (time 1764937796.72 total 10.01 last 68.76) - open/close 1412 (time 1764937806.72 total 20.02 last 72.33) - open/close 2128 (time 1764937816.73 total 30.02 last 71.54) - open/close 2845 (time 1764937826.74 total 40.03 last 71.63) - open/close 3482 (time 1764937836.76 total 50.05 last 63.60) - open/close 4143 (time 1764937846.77 total 60.06 last 66.05) - open/close 4768 (time 1764937856.77 total 70.06 last 62.45) - open/close 5448 (time 1764937866.78 total 80.08 last 67.92) - open/close 6125 (time 1764937876.80 total 90.09 last 67.62) - open/close 6798 (time 1764937886.80 total 100.09 last 67.29) - open/close 7406 (time 1764937896.83 total 110.12 last 60.61) - open/close 7966 (time 1764937906.84 total 120.13 last 55.94) - open/close 8614 (time 1764937916.85 total 130.14 last 64.72) - open/close 9327 (time 1764937926.86 total 140.15 last 71.25) total: 10000 open/close in 150.13 seconds: 66.61 ops/second - unlinked 0 (time 1764937940 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 715 (time 1764938032.45 total 10.01 last 71.44) - open/close 1411 (time 1764938042.46 total 20.02 last 69.54) - open/close 2144 (time 1764938052.47 total 30.02 last 73.25) - open/close 2850 (time 1764938062.47 total 40.03 last 70.58) - open/close 3570 (time 1764938072.48 total 50.04 last 71.93) - open/close 4295 (time 1764938082.49 total 60.05 last 72.43) - open/close 5075 (time 1764938092.50 total 70.05 last 77.94) - open/close 5767 (time 1764938102.50 total 80.06 last 69.17) - open/close 6499 (time 1764938112.51 total 90.06 last 73.15) - open/close 7255 (time 1764938122.51 total 100.06 last 75.60) - open/close 7950 (time 1764938132.51 total 110.07 last 69.48) - open/close 8694 (time 1764938142.52 total 120.08 last 74.34) - open/close 9398 (time 1764938152.54 total 130.10 last 70.25) total: 10000 open/close in 138.97 seconds: 71.96 ops/second - unlinked 0 (time 1764938165 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 725 (time 1764938251.24 total 10.00 last 72.48) - open/close 1385 (time 1764938261.24 total 20.00 last 65.99) - open/close 2075 (time 1764938271.25 total 30.01 last 68.95) - open/close 2750 (time 1764938281.27 total 40.03 last 67.38) - open/close 3418 (time 1764938291.28 total 50.04 last 66.74) - open/close 4108 (time 1764938301.29 total 60.05 last 68.92) - open/close 4836 (time 1764938311.30 total 70.06 last 72.71) - open/close 5544 (time 1764938321.31 total 80.07 last 70.75) - open/close 6311 (time 1764938331.31 total 90.07 last 76.67) - open/close 7058 (time 1764938341.31 total 100.07 last 74.69) - open/close 7790 (time 1764938351.32 total 110.08 last 73.15) - open/close 8571 (time 1764938361.33 total 120.09 last 78.03) - open/close 9326 (time 1764938371.33 total 130.09 last 75.49) total: 10000 open/close in 139.42 seconds: 71.73 ops/second - unlinked 0 (time 1764938384 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 731 (time 1764938484.79 total 10.00 last 73.10) - open/close 1452 (time 1764938494.80 total 20.01 last 72.03) - open/close 2191 (time 1764938504.82 total 30.03 last 73.75) - open/close 2878 (time 1764938514.82 total 40.03 last 68.69) - open/close 3512 (time 1764938524.82 total 50.04 last 63.36) - open/close 4095 (time 1764938534.83 total 60.04 last 58.27) - open/close 4709 (time 1764938544.84 total 70.05 last 61.36) - open/close 5430 (time 1764938554.85 total 80.06 last 72.01) - open/close 6129 (time 1764938564.86 total 90.07 last 69.84) - open/close 6919 (time 1764938574.87 total 100.08 last 78.90) - open/close 7622 (time 1764938584.88 total 110.10 last 70.19) - open/close 8355 (time 1764938594.89 total 120.10 last 73.29) - open/close 9072 (time 1764938604.89 total 130.10 last 71.67) - open/close 9730 (time 1764938614.90 total 140.11 last 65.73) total: 10000 open/close in 144.17 seconds: 69.36 ops/second - unlinked 0 (time 1764938622 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 672 (time 1764938719.45 total 10.01 last 67.13) - open/close 1356 (time 1764938729.46 total 20.02 last 68.35) - open/close 2068 (time 1764938739.47 total 30.03 last 71.09) - open/close 2744 (time 1764938749.47 total 40.04 last 67.59) - open/close 3439 (time 1764938759.48 total 50.04 last 69.49) - open/close 4152 (time 1764938769.48 total 60.04 last 71.29) - open/close 4875 (time 1764938779.49 total 70.05 last 72.23) - open/close 5594 (time 1764938789.49 total 80.06 last 71.85) - open/close 6351 (time 1764938799.50 total 90.06 last 75.66) - open/close 7119 (time 1764938809.51 total 100.07 last 76.76) - open/close 7855 (time 1764938819.51 total 110.08 last 73.54) - open/close 8588 (time 1764938829.52 total 120.08 last 73.29) - open/close 9302 (time 1764938839.53 total 130.09 last 71.30) total: 10000 open/close in 139.69 seconds: 71.59 ops/second - unlinked 0 (time 1764938852 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 817 (time 1764938940.79 total 10.01 last 81.62) - open/close 1660 (time 1764938950.81 total 20.03 last 84.12) - open/close 2430 (time 1764938960.81 total 30.03 last 76.99) - open/close 3165 (time 1764938970.83 total 40.05 last 73.39) - open/close 3850 (time 1764938980.83 total 50.05 last 68.47) - open/close 4544 (time 1764938990.83 total 60.05 last 69.39) - open/close 5286 (time 1764939000.84 total 70.06 last 74.19) - open/close 5893 (time 1764939010.85 total 80.07 last 60.61) - open/close 6559 (time 1764939020.86 total 90.08 last 66.53) - open/close 7229 (time 1764939030.87 total 100.09 last 66.93) - open/close 7948 (time 1764939040.87 total 110.09 last 71.90) - open/close 8629 (time 1764939050.89 total 120.10 last 68.02) - open/close 9353 (time 1764939060.90 total 130.12 last 72.31) total: 10000 open/close in 139.17 seconds: 71.85 ops/second - unlinked 0 (time 1764939073 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 711 (time 1764939165.33 total 10.01 last 71.03) - open/close 1412 (time 1764939175.34 total 20.02 last 70.04) - open/close 2093 (time 1764939185.34 total 30.02 last 68.05) - open/close 2787 (time 1764939195.35 total 40.03 last 69.34) - open/close 3435 (time 1764939205.35 total 50.04 last 64.77) - open/close 4168 (time 1764939215.37 total 60.05 last 73.21) - open/close 4952 (time 1764939225.37 total 70.05 last 78.37) - open/close 5618 (time 1764939235.39 total 80.07 last 66.46) - open/close 6280 (time 1764939245.40 total 90.08 last 66.15) - open/close 6988 (time 1764939255.40 total 100.08 last 70.78) - open/close 7721 (time 1764939265.41 total 110.09 last 73.24) - open/close 8432 (time 1764939275.42 total 120.10 last 71.06) - open/close 9149 (time 1764939285.42 total 130.10 last 71.67) - open/close 9887 (time 1764939295.42 total 140.10 last 73.80) total: 10000 open/close in 141.40 seconds: 70.72 ops/second - unlinked 0 (time 1764939299 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg259-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock oleg259-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg259-server: LNET ready to unload oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2