== conf-sanity test 69: replace an OST with the same index ========================================================== 11:02:13 (1761318133) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg243-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99904. rc=0 - open/close 791 (time 1761318211.78 total 10.01 last 79.04) - open/close 1597 (time 1761318221.78 total 20.01 last 80.56) - open/close 2458 (time 1761318231.80 total 30.02 last 86.00) - open/close 3243 (time 1761318241.80 total 40.03 last 78.44) - open/close 3947 (time 1761318251.81 total 50.04 last 70.37) - open/close 4744 (time 1761318261.82 total 60.05 last 79.61) - open/close 5648 (time 1761318271.82 total 70.05 last 90.38) - open/close 6581 (time 1761318281.82 total 80.05 last 93.29) - open/close 7409 (time 1761318291.84 total 90.07 last 82.68) - open/close 8197 (time 1761318301.85 total 100.08 last 78.72) - open/close 8983 (time 1761318311.85 total 110.08 last 78.60) - open/close 9873 (time 1761318321.86 total 120.08 last 88.94) total: 10000 open/close in 121.42 seconds: 82.36 ops/second - unlinked 0 (time 1761318325 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 871 (time 1761318401.30 total 10.00 last 87.09) - open/close 1660 (time 1761318411.31 total 20.01 last 78.85) - open/close 2416 (time 1761318421.31 total 30.01 last 75.56) - open/close 3259 (time 1761318431.34 total 40.03 last 84.12) - open/close 4082 (time 1761318441.34 total 50.04 last 82.25) - open/close 4929 (time 1761318451.35 total 60.05 last 84.66) - open/close 5737 (time 1761318461.35 total 70.05 last 80.78) - open/close 6676 (time 1761318471.36 total 80.06 last 93.79) - open/close 7487 (time 1761318481.36 total 90.06 last 81.07) - open/close 8413 (time 1761318491.37 total 100.07 last 92.54) - open/close 9241 (time 1761318501.38 total 110.08 last 82.74) total: 10000 open/close in 118.74 seconds: 84.22 ops/second - unlinked 0 (time 1761318512 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1010 (time 1761318588.46 total 10.00 last 100.95) - open/close 1901 (time 1761318598.46 total 20.01 last 89.10) - open/close 2737 (time 1761318608.47 total 30.01 last 83.52) - open/close 3656 (time 1761318618.47 total 40.02 last 91.89) - open/close 4595 (time 1761318628.48 total 50.02 last 93.84) - open/close 5409 (time 1761318638.49 total 60.03 last 81.32) - open/close 6231 (time 1761318648.49 total 70.04 last 82.14) - open/close 7151 (time 1761318658.51 total 80.05 last 91.90) - open/close 8071 (time 1761318668.51 total 90.06 last 91.95) - open/close 8899 (time 1761318678.51 total 100.06 last 82.79) - open/close 9782 (time 1761318688.51 total 110.06 last 88.30) total: 10000 open/close in 112.10 seconds: 89.20 ops/second - unlinked 0 (time 1761318693 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1326 (time 1761318758.01 total 10.00 last 132.54) - open/close 2332 (time 1761318768.02 total 20.01 last 100.52) - open/close 3280 (time 1761318778.02 total 30.01 last 94.79) - open/close 4156 (time 1761318788.03 total 40.02 last 87.53) - open/close 5284 (time 1761318798.04 total 50.03 last 112.71) - open/close 6486 (time 1761318808.04 total 60.03 last 120.17) - open/close 7637 (time 1761318818.05 total 70.04 last 115.04) - open/close 9376 (time 1761318828.06 total 80.05 last 173.68) total: 10000 open/close in 84.66 seconds: 118.12 ops/second - unlinked 0 (time 1761318834 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1025 (time 1761318896.63 total 10.01 last 102.44) - open/close 2081 (time 1761318906.64 total 20.02 last 105.50) - open/close 3144 (time 1761318916.65 total 30.02 last 106.24) - open/close 4377 (time 1761318926.66 total 40.03 last 123.18) - open/close 5496 (time 1761318936.66 total 50.03 last 111.89) - open/close 6495 (time 1761318946.66 total 60.04 last 99.84) - open/close 7516 (time 1761318956.67 total 70.04 last 102.05) - open/close 8701 (time 1761318966.67 total 80.04 last 118.48) total: 10000 open/close in 89.79 seconds: 111.37 ops/second - unlinked 0 (time 1761318978 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1130 (time 1761319035.75 total 10.01 last 112.94) - open/close 2261 (time 1761319045.75 total 20.01 last 113.08) - open/close 3275 (time 1761319055.76 total 30.01 last 101.34) - open/close 4600 (time 1761319065.76 total 40.02 last 132.44) - open/close 5692 (time 1761319075.77 total 50.02 last 109.14) - open/close 6651 (time 1761319085.77 total 60.03 last 95.88) - open/close 7843 (time 1761319095.77 total 70.03 last 119.18) - open/close 9314 (time 1761319105.78 total 80.04 last 146.97) total: 10000 open/close in 84.30 seconds: 118.63 ops/second - unlinked 0 (time 1761319111 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1424 (time 1761319165.79 total 10.01 last 142.26) - open/close 2886 (time 1761319175.80 total 20.01 last 146.15) - open/close 4443 (time 1761319185.80 total 30.02 last 155.64) - open/close 5963 (time 1761319195.80 total 40.02 last 151.95) - open/close 7619 (time 1761319205.81 total 50.02 last 165.54) - open/close 9305 (time 1761319215.81 total 60.02 last 168.59) total: 10000 open/close in 64.25 seconds: 155.63 ops/second - unlinked 0 (time 1761319221 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1610 (time 1761319265.09 total 10.01 last 160.92) - open/close 3171 (time 1761319275.09 total 20.01 last 156.06) - open/close 4488 (time 1761319285.09 total 30.01 last 131.69) - open/close 5945 (time 1761319295.09 total 40.01 last 145.63) - open/close 7091 (time 1761319305.10 total 50.01 last 114.59) - open/close 8333 (time 1761319315.10 total 60.02 last 124.15) - open/close 9678 (time 1761319325.11 total 70.02 last 134.43) total: 10000 open/close in 72.71 seconds: 137.54 ops/second - unlinked 0 (time 1761319330 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1870 (time 1761319375.89 total 10.00 last 186.91) - open/close 3735 (time 1761319385.90 total 20.01 last 186.48) - open/close 5952 (time 1761319395.90 total 30.01 last 221.69) - open/close 7797 (time 1761319405.90 total 40.01 last 184.47) - open/close 9802 (time 1761319415.90 total 50.01 last 200.46) total: 10000 open/close in 51.08 seconds: 195.76 ops/second - unlinked 0 (time 1761319418 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1592 (time 1761319460.74 total 10.00 last 159.13) - open/close 3334 (time 1761319470.74 total 20.01 last 174.15) - open/close 5600 (time 1761319480.74 total 30.01 last 226.54) - open/close 7534 (time 1761319490.75 total 40.01 last 193.36) - open/close 9023 (time 1761319500.75 total 50.02 last 148.80) total: 10000 open/close in 55.60 seconds: 179.87 ops/second - unlinked 0 (time 1761319507 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg243-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg243-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg243-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg243-server: LNET ready to unload oleg243-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2