== conf-sanity test 69: replace an OST with the same index ========================================================== 09:17:04 (1755695824) start mds service on oleg304-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions oleg304-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg304-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg304-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 2495 (time 1755695884.51 total 10.00 last 249.45) - open/close 4818 (time 1755695894.51 total 20.00 last 232.29) - open/close 7387 (time 1755695904.51 total 30.00 last 256.84) total: 10000 open/close in 39.55 seconds: 252.83 ops/second - unlinked 0 (time 1755695915 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2249 (time 1755695948.62 total 10.00 last 224.88) - open/close 6468 (time 1755695958.63 total 20.00 last 421.82) - open/close 9851 (time 1755695968.63 total 30.00 last 338.28) total: 10000 open/close in 30.59 seconds: 326.94 ops/second - unlinked 0 (time 1755695970 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2399 (time 1755696000.41 total 10.00 last 239.86) - open/close 4665 (time 1755696010.41 total 20.00 last 226.57) - open/close 6938 (time 1755696020.41 total 30.01 last 227.23) - open/close 9277 (time 1755696030.41 total 40.01 last 233.89) total: 10000 open/close in 43.02 seconds: 232.47 ops/second - unlinked 0 (time 1755696034 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2429 (time 1755696067.32 total 10.00 last 242.86) - open/close 4778 (time 1755696077.32 total 20.00 last 234.86) - open/close 7152 (time 1755696087.33 total 30.01 last 237.33) total: 10000 open/close in 39.66 seconds: 252.15 ops/second - unlinked 0 (time 1755696098 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2739 (time 1755696129.78 total 10.00 last 273.89) - open/close 5599 (time 1755696139.78 total 20.00 last 285.96) - open/close 8111 (time 1755696149.79 total 30.00 last 251.16) total: 10000 open/close in 36.97 seconds: 270.50 ops/second - unlinked 0 (time 1755696158 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 3041 (time 1755696193.61 total 10.00 last 304.04) - open/close 5584 (time 1755696203.62 total 20.00 last 254.22) - open/close 8263 (time 1755696213.62 total 30.01 last 267.88) total: 10000 open/close in 35.97 seconds: 278.03 ops/second - unlinked 0 (time 1755696221 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3121 (time 1755696252.74 total 10.00 last 311.99) - open/close 5868 (time 1755696262.74 total 20.01 last 274.60) - open/close 8601 (time 1755696272.75 total 30.01 last 273.18) total: 10000 open/close in 35.89 seconds: 278.65 ops/second - unlinked 0 (time 1755696280 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2954 (time 1755696310.36 total 10.00 last 295.35) - open/close 5749 (time 1755696320.36 total 20.00 last 279.47) - open/close 8637 (time 1755696330.36 total 30.00 last 288.77) total: 10000 open/close in 34.14 seconds: 292.92 ops/second - unlinked 0 (time 1755696336 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2739 (time 1755696369.54 total 10.00 last 273.88) - open/close 5313 (time 1755696379.54 total 20.00 last 257.31) - open/close 9366 (time 1755696389.54 total 30.01 last 405.25) total: 10000 open/close in 31.25 seconds: 320.00 ops/second - unlinked 0 (time 1755696392 ; total 1 ; last 1) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 3018 (time 1755696415.58 total 10.00 last 301.69) - open/close 6085 (time 1755696425.58 total 20.01 last 306.62) - open/close 9159 (time 1755696435.58 total 30.01 last 307.37) total: 10000 open/close in 33.03 seconds: 302.76 ops/second - unlinked 0 (time 1755696440 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg304-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg304-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre On OST0, 10485 used inodes rc=0 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.104@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg304-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg304-server: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg304-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg304-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2