== conf-sanity test 69: replace an OST with the same index ========================================================== 12:09:43 (1761322183) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg428-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock oleg428-server@tcp:/lustre /mnt/lustre On OST0, 49700 inodes available. Want 99872. rc=0 - open/close 1699 (time 1761322232.65 total 10.00 last 169.83) - open/close 3537 (time 1761322242.65 total 20.01 last 183.78) - open/close 5673 (time 1761322252.65 total 30.01 last 213.54) - open/close 7867 (time 1761322262.66 total 40.02 last 219.24) total: 10000 open/close in 49.91 seconds: 200.34 ops/second - unlinked 0 (time 1761322274 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2081 (time 1761322308.61 total 10.00 last 208.00) - open/close 4254 (time 1761322318.61 total 20.01 last 217.29) - open/close 6131 (time 1761322328.62 total 30.01 last 187.62) - open/close 8576 (time 1761322338.62 total 40.01 last 244.49) total: 10000 open/close in 45.06 seconds: 221.94 ops/second - unlinked 0 (time 1761322345 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2010 (time 1761322382.43 total 10.00 last 200.97) - open/close 4221 (time 1761322392.43 total 20.00 last 221.10) - open/close 6997 (time 1761322402.43 total 30.00 last 277.56) - open/close 9848 (time 1761322412.43 total 40.00 last 285.06) total: 10000 open/close in 40.56 seconds: 246.57 ops/second - unlinked 0 (time 1761322414 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2399 (time 1761322451.20 total 10.00 last 239.78) - open/close 4467 (time 1761322461.21 total 20.01 last 206.70) - open/close 6577 (time 1761322471.21 total 30.01 last 210.97) - open/close 8685 (time 1761322481.21 total 40.01 last 210.76) total: 10000 open/close in 47.36 seconds: 211.13 ops/second - unlinked 0 (time 1761322490 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 3478 (time 1761322525.64 total 10.00 last 347.79) - open/close 6266 (time 1761322535.64 total 20.00 last 278.76) - open/close 8622 (time 1761322545.64 total 30.00 last 235.53) total: 10000 open/close in 34.38 seconds: 290.84 ops/second - unlinked 0 (time 1761322551 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 2999 (time 1761322579.10 total 10.00 last 299.84) - open/close 6006 (time 1761322589.10 total 20.00 last 300.62) - open/close 9844 (time 1761322599.10 total 30.00 last 383.78) total: 10000 open/close in 30.38 seconds: 329.15 ops/second - unlinked 0 (time 1761322600 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3602 (time 1761322628.53 total 10.00 last 360.20) - open/close 7453 (time 1761322638.53 total 20.00 last 385.08) total: 10000 open/close in 26.20 seconds: 381.74 ops/second - unlinked 0 (time 1761322646 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4232 (time 1761322670.94 total 10.00 last 423.07) - open/close 7827 (time 1761322680.94 total 20.00 last 359.49) total: 10000 open/close in 25.59 seconds: 390.70 ops/second - unlinked 0 (time 1761322687 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 4047 (time 1761322714.73 total 10.00 last 404.61) - open/close 8213 (time 1761322724.73 total 20.00 last 416.59) total: 10000 open/close in 23.56 seconds: 424.49 ops/second - unlinked 0 (time 1761322729 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4344 (time 1761322751.61 total 10.00 last 434.40) - open/close 8227 (time 1761322761.61 total 20.00 last 388.28) total: 10000 open/close in 24.04 seconds: 415.90 ops/second - unlinked 0 (time 1761322767 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.128@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg428-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock oleg428-server@tcp:/lustre /mnt/lustre On OST0, 10235 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg428-server: LNET ready to unload oleg428-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2