== conf-sanity test 69: replace an OST with the same index ========================================================== 23:53:20 (1790740400) start mds service on oleg234-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: kmemleak disabled oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg234-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 858 (time 1790740528.60 total 10.00 last 85.78) - open/close 1511 (time 1790740538.61 total 20.01 last 65.23) - open/close 2231 (time 1790740548.64 total 30.04 last 71.81) - open/close 3042 (time 1790740558.64 total 40.05 last 81.04) - open/close 3976 (time 1790740568.64 total 50.05 last 93.40) - open/close 5136 (time 1790740578.65 total 60.05 last 115.98) - open/close 5980 (time 1790740588.65 total 70.05 last 84.38) - open/close 6734 (time 1790740598.65 total 80.06 last 75.35) - open/close 7442 (time 1790740608.67 total 90.07 last 70.73) - open/close 8148 (time 1790740618.69 total 100.09 last 70.45) - open/close 8843 (time 1790740628.69 total 110.09 last 69.48) - open/close 9745 (time 1790740638.70 total 120.10 last 90.09) total: 10000 open/close in 123.12 seconds: 81.22 ops/second - unlinked 0 (time 1790740646 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 984 (time 1790740730.70 total 10.00 last 98.37) - open/close 1860 (time 1790740740.70 total 20.01 last 87.54) - open/close 2763 (time 1790740750.71 total 30.02 last 90.24) - open/close 3641 (time 1790740760.72 total 40.03 last 87.68) - open/close 4504 (time 1790740770.73 total 50.04 last 86.23) - open/close 5246 (time 1790740780.77 total 60.08 last 73.90) - open/close 6087 (time 1790740790.80 total 70.11 last 83.85) - open/close 6651 (time 1790740800.82 total 80.12 last 56.31) - open/close 7302 (time 1790740810.82 total 90.13 last 65.05) - open/close 8202 (time 1790740820.83 total 100.13 last 89.98) - open/close 8919 (time 1790740830.84 total 110.15 last 71.61) - open/close 9792 (time 1790740840.84 total 120.15 last 87.29) total: 10000 open/close in 122.20 seconds: 81.83 ops/second - unlinked 0 (time 1790740847 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1014 (time 1790740922.96 total 10.00 last 101.37) - open/close 1957 (time 1790740932.96 total 20.01 last 94.27) - open/close 2789 (time 1790740942.96 total 30.01 last 83.17) - open/close 3516 (time 1790740952.97 total 40.01 last 72.66) - open/close 4073 (time 1790740962.98 total 50.02 last 55.65) - open/close 4999 (time 1790740972.98 total 60.03 last 92.56) - open/close 5828 (time 1790740982.99 total 70.03 last 82.84) - open/close 6637 (time 1790740992.99 total 80.04 last 80.88) - open/close 7507 (time 1790741002.99 total 90.04 last 86.98) - open/close 8368 (time 1790741013.00 total 100.04 last 86.06) - open/close 9334 (time 1790741023.00 total 110.05 last 96.53) total: 10000 open/close in 116.99 seconds: 85.47 ops/second - unlinked 0 (time 1790741034 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 891 (time 1790741125.50 total 10.01 last 88.98) - open/close 1727 (time 1790741135.52 total 20.03 last 83.49) - open/close 2593 (time 1790741145.53 total 30.04 last 86.44) - open/close 3391 (time 1790741155.54 total 40.05 last 79.76) - open/close 4373 (time 1790741165.55 total 50.06 last 98.14) - open/close 5309 (time 1790741175.55 total 60.06 last 93.59) - open/close 6275 (time 1790741185.55 total 70.06 last 96.59) - open/close 7050 (time 1790741195.56 total 80.07 last 77.44) - open/close 7771 (time 1790741205.56 total 90.07 last 72.07) - open/close 8499 (time 1790741215.56 total 100.07 last 72.78) - open/close 9291 (time 1790741225.56 total 110.08 last 79.18) total: 10000 open/close in 118.14 seconds: 84.65 ops/second - unlinked 0 (time 1790741238 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 865 (time 1790741320.09 total 10.00 last 86.48) - open/close 1783 (time 1790741330.09 total 20.01 last 91.75) - open/close 2657 (time 1790741340.10 total 30.02 last 87.32) - open/close 3559 (time 1790741350.11 total 40.02 last 90.18) - open/close 4442 (time 1790741360.11 total 50.02 last 88.26) - open/close 5369 (time 1790741370.11 total 60.03 last 92.69) - open/close 6435 (time 1790741380.12 total 70.03 last 106.57) - open/close 7503 (time 1790741390.12 total 80.03 last 106.78) - open/close 8428 (time 1790741400.13 total 90.04 last 92.40) - open/close 9332 (time 1790741410.14 total 100.06 last 90.27) total: 10000 open/close in 106.82 seconds: 93.62 ops/second - unlinked 0 (time 1790741421 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 849 (time 1790741494.78 total 10.01 last 84.85) - open/close 1735 (time 1790741504.78 total 20.01 last 88.54) - open/close 2686 (time 1790741514.79 total 30.02 last 95.03) - open/close 3742 (time 1790741524.80 total 40.03 last 105.49) - open/close 4574 (time 1790741534.81 total 50.03 last 83.16) - open/close 5394 (time 1790741544.81 total 60.04 last 81.95) - open/close 6316 (time 1790741554.81 total 70.04 last 92.18) - open/close 7182 (time 1790741564.84 total 80.07 last 86.36) - open/close 8044 (time 1790741574.85 total 90.08 last 86.14) - open/close 8991 (time 1790741584.85 total 100.08 last 94.66) - open/close 9870 (time 1790741594.86 total 110.09 last 87.82) total: 10000 open/close in 111.37 seconds: 89.79 ops/second - unlinked 0 (time 1790741600 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 730 (time 1790741680.99 total 10.01 last 72.96) - open/close 1655 (time 1790741690.99 total 20.01 last 92.48) - open/close 2649 (time 1790741701.00 total 30.01 last 99.38) - open/close 3601 (time 1790741711.00 total 40.01 last 95.19) - open/close 4532 (time 1790741721.00 total 50.02 last 93.05) - open/close 5493 (time 1790741731.01 total 60.02 last 96.04) - open/close 6428 (time 1790741741.01 total 70.03 last 93.44) - open/close 7454 (time 1790741751.02 total 80.03 last 102.60) - open/close 8430 (time 1790741761.02 total 90.03 last 97.57) - open/close 9416 (time 1790741771.02 total 100.03 last 98.58) total: 10000 open/close in 105.77 seconds: 94.54 ops/second - unlinked 0 (time 1790741780 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 959 (time 1790741856.69 total 10.01 last 95.84) - open/close 2017 (time 1790741866.70 total 20.02 last 105.69) - open/close 3081 (time 1790741876.71 total 30.02 last 106.38) - open/close 4003 (time 1790741886.71 total 40.03 last 92.14) - open/close 5019 (time 1790741896.71 total 50.03 last 101.57) - open/close 6058 (time 1790741906.72 total 60.03 last 103.88) - open/close 6775 (time 1790741916.72 total 70.04 last 71.66) - open/close 7627 (time 1790741926.74 total 80.05 last 85.09) - open/close 8595 (time 1790741936.74 total 90.05 last 96.79) - open/close 9479 (time 1790741946.74 total 100.06 last 88.35) total: 10000 open/close in 106.15 seconds: 94.21 ops/second - unlinked 0 (time 1790741957 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 759 (time 1790742031.74 total 10.01 last 75.86) - open/close 1436 (time 1790742041.74 total 20.01 last 67.65) - open/close 2264 (time 1790742051.75 total 30.02 last 82.74) - open/close 3136 (time 1790742061.78 total 40.05 last 86.95) - open/close 4051 (time 1790742071.79 total 50.05 last 91.44) - open/close 5064 (time 1790742081.79 total 60.06 last 101.29) - open/close 5982 (time 1790742091.80 total 70.07 last 91.65) - open/close 6937 (time 1790742101.80 total 80.07 last 95.49) - open/close 7820 (time 1790742111.81 total 90.08 last 88.24) - open/close 8782 (time 1790742121.82 total 100.09 last 96.13) - open/close 9836 (time 1790742131.82 total 110.09 last 105.40) total: 10000 open/close in 111.44 seconds: 89.73 ops/second - unlinked 0 (time 1790742136 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1032 (time 1790742215.88 total 10.01 last 103.10) - open/close 2124 (time 1790742225.88 total 20.01 last 109.17) - open/close 2994 (time 1790742235.88 total 30.02 last 86.98) - open/close 4018 (time 1790742245.89 total 40.02 last 102.35) - open/close 4996 (time 1790742255.89 total 50.03 last 97.75) - open/close 5950 (time 1790742265.90 total 60.03 last 95.33) - open/close 7083 (time 1790742275.91 total 70.04 last 113.24) - open/close 8150 (time 1790742285.91 total 80.04 last 106.70) - open/close 9180 (time 1790742295.91 total 90.04 last 102.99) total: 10000 open/close in 96.91 seconds: 103.19 ops/second - unlinked 0 (time 1790742306 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg234-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg234-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.134@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.134@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.134@tcp sys.timeout=20 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg234-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg234-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: kmemleak disabled modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2