== conf-sanity test 69: replace an OST with the same index ========================================================== 20:07:18 (1777421238) start mds service on oleg447-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg447-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg447-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49535 inodes available. Want 99872. rc=0 - open/close 1250 (time 1777421302.26 total 10.00 last 124.99) - open/close 2333 (time 1777421312.27 total 20.01 last 108.20) - open/close 3314 (time 1777421322.28 total 30.01 last 98.07) - open/close 4367 (time 1777421332.28 total 40.02 last 105.27) - open/close 5461 (time 1777421342.29 total 50.02 last 109.32) - open/close 6527 (time 1777421352.29 total 60.02 last 106.59) - open/close 7592 (time 1777421362.30 total 70.03 last 106.40) - open/close 8698 (time 1777421372.30 total 80.04 last 110.57) - open/close 9759 (time 1777421382.30 total 90.04 last 106.04) total: 10000 open/close in 92.54 seconds: 108.06 ops/second - unlinked 0 (time 1777421387 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1188 (time 1777421455.21 total 10.00 last 118.78) - open/close 2431 (time 1777421465.22 total 20.00 last 124.27) - open/close 3849 (time 1777421475.22 total 30.01 last 141.69) - open/close 5226 (time 1777421485.23 total 40.01 last 137.66) - open/close 6724 (time 1777421495.23 total 50.02 last 149.76) - open/close 7984 (time 1777421505.23 total 60.02 last 125.95) - open/close 9259 (time 1777421515.24 total 70.03 last 127.41) total: 10000 open/close in 75.75 seconds: 132.00 ops/second - unlinked 0 (time 1777421522 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1321 (time 1777421580.13 total 10.01 last 131.98) - open/close 2557 (time 1777421590.13 total 20.01 last 123.59) - open/close 3997 (time 1777421600.14 total 30.01 last 143.92) - open/close 6015 (time 1777421610.14 total 40.02 last 201.68) - open/close 8018 (time 1777421620.14 total 50.02 last 200.26) - open/close 9890 (time 1777421630.15 total 60.02 last 187.18) total: 10000 open/close in 60.58 seconds: 165.06 ops/second - unlinked 0 (time 1777421632 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1675 (time 1777421672.85 total 10.00 last 167.43) - open/close 3473 (time 1777421682.86 total 20.01 last 179.77) - open/close 5080 (time 1777421692.86 total 30.01 last 160.64) - open/close 6961 (time 1777421702.86 total 40.01 last 188.10) - open/close 8703 (time 1777421712.86 total 50.01 last 174.16) total: 10000 open/close in 57.52 seconds: 173.86 ops/second - unlinked 0 (time 1777421721 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1753 (time 1777421760.96 total 10.00 last 175.29) - open/close 3612 (time 1777421770.96 total 20.00 last 185.89) - open/close 5746 (time 1777421780.96 total 30.00 last 213.38) - open/close 8287 (time 1777421790.97 total 40.01 last 253.87) - open/close 9194 (time 1777421800.97 total 50.01 last 90.69) total: 10000 open/close in 57.46 seconds: 174.03 ops/second - unlinked 0 (time 1777421811 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 970 (time 1777421897.08 total 10.01 last 96.95) - open/close 1962 (time 1777421907.08 total 20.01 last 99.18) - open/close 2899 (time 1777421917.10 total 30.02 last 93.56) - open/close 3780 (time 1777421927.10 total 40.03 last 88.06) - open/close 4605 (time 1777421937.10 total 50.03 last 82.49) - open/close 5531 (time 1777421947.11 total 60.04 last 92.51) - open/close 6498 (time 1777421957.12 total 70.04 last 96.63) - open/close 7409 (time 1777421967.13 total 80.05 last 91.05) - open/close 8317 (time 1777421977.13 total 90.05 last 90.77) - open/close 9392 (time 1777421987.14 total 100.06 last 107.40) total: 10000 open/close in 106.04 seconds: 94.30 ops/second - unlinked 0 (time 1777421995 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 757 (time 1777422075.75 total 10.02 last 75.56) - open/close 1675 (time 1777422085.75 total 20.02 last 91.77) - open/close 2893 (time 1777422095.75 total 30.02 last 121.80) - open/close 4319 (time 1777422105.76 total 40.03 last 142.52) - open/close 5788 (time 1777422115.76 total 50.03 last 146.85) - open/close 7033 (time 1777422125.76 total 60.03 last 124.49) - open/close 8978 (time 1777422135.77 total 70.03 last 194.45) total: 10000 open/close in 74.35 seconds: 134.49 ops/second - unlinked 0 (time 1777422141 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 1995 (time 1777422175.80 total 10.01 last 199.37) - open/close 3856 (time 1777422185.80 total 20.01 last 186.01) - open/close 5561 (time 1777422195.80 total 30.01 last 170.50) - open/close 7317 (time 1777422205.81 total 40.02 last 175.49) - open/close 9077 (time 1777422215.81 total 50.02 last 175.94) total: 10000 open/close in 54.52 seconds: 183.43 ops/second - unlinked 0 (time 1777422222 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2178 (time 1777422262.93 total 10.00 last 217.77) - open/close 4601 (time 1777422272.93 total 20.01 last 242.20) - open/close 7343 (time 1777422282.94 total 30.01 last 274.10) - open/close 9615 (time 1777422292.94 total 40.02 last 227.06) total: 10000 open/close in 41.87 seconds: 238.86 ops/second - unlinked 0 (time 1777422296 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3224 (time 1777422326.88 total 10.00 last 322.40) - open/close 5757 (time 1777422336.88 total 20.00 last 253.26) - open/close 9124 (time 1777422346.88 total 30.00 last 336.66) total: 10000 open/close in 32.20 seconds: 310.59 ops/second - unlinked 0 (time 1777422350 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server unloading modules via unload_modules_local on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local oleg447-server: modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2