== conf-sanity test 69: replace an OST with the same index ========================================================== 09:42:46 (1761313366) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: detected 4 online CPUs by sysfs oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg218-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg218-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock oleg218-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 738 (time 1761313464.77 total 10.00 last 73.80) - open/close 1483 (time 1761313474.78 total 20.00 last 74.47) - open/close 2303 (time 1761313484.78 total 30.01 last 81.95) - open/close 3050 (time 1761313494.78 total 40.01 last 74.68) - open/close 3770 (time 1761313504.81 total 50.03 last 71.85) - open/close 4454 (time 1761313514.82 total 60.05 last 68.31) - open/close 5175 (time 1761313524.82 total 70.05 last 72.08) - open/close 5853 (time 1761313534.82 total 80.05 last 67.77) - open/close 6617 (time 1761313544.84 total 90.07 last 76.27) - open/close 7370 (time 1761313554.85 total 100.07 last 75.27) - open/close 8131 (time 1761313564.85 total 110.08 last 76.05) - open/close 8906 (time 1761313574.85 total 120.08 last 77.48) - open/close 9715 (time 1761313584.86 total 130.09 last 80.87) total: 10000 open/close in 133.87 seconds: 74.70 ops/second - unlinked 0 (time 1761313592 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 751 (time 1761313689.90 total 10.00 last 75.07) - open/close 1511 (time 1761313699.90 total 20.01 last 75.98) - open/close 2378 (time 1761313709.91 total 30.01 last 86.64) - open/close 3276 (time 1761313719.92 total 40.02 last 89.76) - open/close 4083 (time 1761313729.92 total 50.02 last 80.69) - open/close 4866 (time 1761313739.93 total 60.03 last 78.19) - open/close 5605 (time 1761313749.93 total 70.04 last 73.88) - open/close 6367 (time 1761313759.94 total 80.04 last 76.16) - open/close 7117 (time 1761313769.94 total 90.04 last 74.98) - open/close 7794 (time 1761313779.94 total 100.05 last 67.68) - open/close 8399 (time 1761313789.96 total 110.06 last 60.42) - open/close 9116 (time 1761313799.97 total 120.08 last 71.59) - open/close 9804 (time 1761313809.98 total 130.08 last 68.75) total: 10000 open/close in 132.73 seconds: 75.34 ops/second - unlinked 0 (time 1761313816 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 753 (time 1761313913.92 total 10.01 last 75.23) - open/close 1459 (time 1761313923.93 total 20.01 last 70.56) - open/close 2213 (time 1761313933.93 total 30.02 last 75.39) - open/close 2978 (time 1761313943.93 total 40.02 last 76.50) - open/close 3716 (time 1761313953.93 total 50.02 last 73.75) - open/close 4477 (time 1761313963.94 total 60.03 last 76.05) - open/close 5265 (time 1761313973.95 total 70.04 last 78.69) - open/close 6067 (time 1761313983.97 total 80.06 last 80.06) - open/close 6800 (time 1761313993.98 total 90.07 last 73.21) - open/close 7545 (time 1761314003.99 total 100.07 last 74.47) - open/close 8255 (time 1761314013.99 total 110.08 last 70.98) - open/close 8993 (time 1761314023.99 total 120.08 last 73.78) - open/close 9705 (time 1761314034.00 total 130.09 last 71.13) total: 10000 open/close in 133.82 seconds: 74.73 ops/second - unlinked 0 (time 1761314041 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 729 (time 1761314134.07 total 10.00 last 72.89) - open/close 1439 (time 1761314144.08 total 20.01 last 70.92) - open/close 2111 (time 1761314154.09 total 30.02 last 67.17) - open/close 2772 (time 1761314164.09 total 40.02 last 66.08) - open/close 3498 (time 1761314174.10 total 50.03 last 72.51) - open/close 4185 (time 1761314184.11 total 60.04 last 68.62) - open/close 4889 (time 1761314194.12 total 70.05 last 70.34) - open/close 5659 (time 1761314204.13 total 80.06 last 76.93) - open/close 6396 (time 1761314214.14 total 90.07 last 73.67) - open/close 7142 (time 1761314224.17 total 100.10 last 74.31) - open/close 7851 (time 1761314234.18 total 110.11 last 70.87) - open/close 8538 (time 1761314244.20 total 120.13 last 68.56) - open/close 9155 (time 1761314254.20 total 130.13 last 61.69) - open/close 9751 (time 1761314264.20 total 140.13 last 59.59) total: 10000 open/close in 143.50 seconds: 69.69 ops/second - unlinked 0 (time 1761314272 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 692 (time 1761314366.87 total 10.00 last 69.17) - open/close 1455 (time 1761314376.87 total 20.01 last 76.29) - open/close 2200 (time 1761314386.88 total 30.01 last 74.43) - open/close 2978 (time 1761314396.88 total 40.02 last 77.78) - open/close 3699 (time 1761314406.89 total 50.03 last 72.04) - open/close 4421 (time 1761314416.89 total 60.03 last 72.18) - open/close 5138 (time 1761314426.90 total 70.04 last 71.65) - open/close 5869 (time 1761314436.90 total 80.04 last 73.08) - open/close 6631 (time 1761314446.91 total 90.04 last 76.18) - open/close 7397 (time 1761314456.92 total 100.05 last 76.53) - open/close 8080 (time 1761314466.93 total 110.06 last 68.23) - open/close 8766 (time 1761314476.93 total 120.06 last 68.59) - open/close 9511 (time 1761314486.94 total 130.08 last 74.39) total: 10000 open/close in 136.72 seconds: 73.14 ops/second - unlinked 0 (time 1761314496 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 654 (time 1761314595.08 total 10.01 last 65.34) - open/close 1433 (time 1761314605.08 total 20.01 last 77.88) - open/close 2226 (time 1761314615.08 total 30.01 last 79.29) - open/close 3012 (time 1761314625.08 total 40.02 last 78.55) - open/close 3760 (time 1761314635.10 total 50.03 last 74.72) - open/close 4475 (time 1761314645.11 total 60.04 last 71.42) - open/close 5192 (time 1761314655.11 total 70.04 last 71.67) - open/close 5934 (time 1761314665.11 total 80.05 last 74.19) - open/close 6723 (time 1761314675.12 total 90.05 last 78.87) - open/close 7470 (time 1761314685.12 total 100.06 last 74.64) - open/close 8167 (time 1761314695.14 total 110.07 last 69.61) - open/close 8881 (time 1761314705.14 total 120.08 last 71.35) - open/close 9584 (time 1761314715.15 total 130.09 last 70.24) total: 10000 open/close in 135.52 seconds: 73.79 ops/second - unlinked 0 (time 1761314724 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 683 (time 1761314816.70 total 10.01 last 68.23) - open/close 1371 (time 1761314826.71 total 20.02 last 68.77) - open/close 2164 (time 1761314836.71 total 30.02 last 79.27) - open/close 2919 (time 1761314846.72 total 40.03 last 75.45) - open/close 3691 (time 1761314856.73 total 50.04 last 77.13) - open/close 4455 (time 1761314866.74 total 60.05 last 76.29) - open/close 5201 (time 1761314876.75 total 70.06 last 74.54) - open/close 5959 (time 1761314886.76 total 80.07 last 75.73) - open/close 6648 (time 1761314896.77 total 90.07 last 68.85) - open/close 7379 (time 1761314906.77 total 100.08 last 73.06) - open/close 8107 (time 1761314916.77 total 110.08 last 72.79) - open/close 8849 (time 1761314926.78 total 120.09 last 74.13) - open/close 9528 (time 1761314936.79 total 130.10 last 67.85) total: 10000 open/close in 136.55 seconds: 73.24 ops/second - unlinked 0 (time 1761314947 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 703 (time 1761315039.13 total 10.01 last 70.22) - open/close 1409 (time 1761315049.14 total 20.02 last 70.55) - open/close 2054 (time 1761315059.14 total 30.02 last 64.48) - open/close 2795 (time 1761315069.16 total 40.04 last 73.98) - open/close 3523 (time 1761315079.17 total 50.05 last 72.75) - open/close 4289 (time 1761315089.18 total 60.06 last 76.52) - open/close 5076 (time 1761315099.19 total 70.07 last 78.61) - open/close 5823 (time 1761315109.19 total 80.07 last 74.70) - open/close 6515 (time 1761315119.20 total 90.08 last 69.13) - open/close 7249 (time 1761315129.21 total 100.09 last 73.29) - open/close 7960 (time 1761315139.22 total 110.10 last 71.05) - open/close 8730 (time 1761315149.23 total 120.11 last 76.93) - open/close 9460 (time 1761315159.23 total 130.11 last 72.99) total: 10000 open/close in 137.08 seconds: 72.95 ops/second - unlinked 0 (time 1761315169 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 728 (time 1761315265.03 total 10.01 last 72.74) - open/close 1418 (time 1761315275.04 total 20.02 last 68.95) - open/close 2023 (time 1761315285.06 total 30.03 last 60.40) - open/close 2542 (time 1761315295.08 total 40.05 last 51.81) - open/close 3201 (time 1761315305.08 total 50.06 last 65.86) - open/close 3893 (time 1761315315.09 total 60.06 last 69.17) - open/close 4560 (time 1761315325.09 total 70.06 last 66.69) - open/close 5240 (time 1761315335.10 total 80.07 last 67.94) - open/close 5901 (time 1761315345.11 total 90.09 last 65.99) - open/close 6519 (time 1761315355.11 total 100.09 last 61.80) - open/close 7198 (time 1761315365.11 total 110.09 last 67.90) - open/close 7934 (time 1761315375.12 total 120.10 last 73.55) - open/close 8679 (time 1761315385.13 total 130.11 last 74.42) - open/close 9386 (time 1761315395.14 total 140.11 last 70.66) total: 10000 open/close in 148.30 seconds: 67.43 ops/second - unlinked 0 (time 1761315407 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 789 (time 1761315501.54 total 10.00 last 78.87) - open/close 1496 (time 1761315511.55 total 20.01 last 70.65) - open/close 2165 (time 1761315521.56 total 30.02 last 66.87) - open/close 2896 (time 1761315531.57 total 40.02 last 73.03) - open/close 3645 (time 1761315541.57 total 50.02 last 74.90) - open/close 4388 (time 1761315551.58 total 60.03 last 74.23) - open/close 5094 (time 1761315561.58 total 70.04 last 70.60) - open/close 5763 (time 1761315571.58 total 80.04 last 66.85) - open/close 6452 (time 1761315581.58 total 90.04 last 68.90) - open/close 7206 (time 1761315591.58 total 100.04 last 75.39) - open/close 7948 (time 1761315601.60 total 110.05 last 74.12) - open/close 8692 (time 1761315611.61 total 120.07 last 74.31) - open/close 9395 (time 1761315621.62 total 130.08 last 70.22) total: 10000 open/close in 137.91 seconds: 72.51 ops/second - unlinked 0 (time 1761315633 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.118@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg218-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg218-server: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg218-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg218-server: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg218-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock oleg218-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg218-server: LNET ready to unload oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2