== conf-sanity test 69: replace an OST with the same index ========================================================== 22:52:10 (1763351530) start mds service on oleg437-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 790 (time 1763351619.03 total 10.01 last 78.95) - open/close 1648 (time 1763351629.04 total 20.02 last 85.68) - open/close 2380 (time 1763351639.05 total 30.03 last 73.14) - open/close 3027 (time 1763351649.06 total 40.03 last 64.67) - open/close 3742 (time 1763351659.06 total 50.04 last 71.47) - open/close 4435 (time 1763351669.07 total 60.05 last 69.21) - open/close 5207 (time 1763351679.08 total 70.06 last 77.13) - open/close 5963 (time 1763351689.10 total 80.08 last 75.46) - open/close 6737 (time 1763351699.10 total 90.08 last 77.38) - open/close 7540 (time 1763351709.11 total 100.09 last 80.24) - open/close 8299 (time 1763351719.11 total 110.09 last 75.88) - open/close 9011 (time 1763351729.12 total 120.10 last 71.15) - open/close 9725 (time 1763351739.12 total 130.10 last 71.38) total: 10000 open/close in 133.44 seconds: 74.94 ops/second - unlinked 0 (time 1763351745 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 819 (time 1763351833.87 total 10.01 last 81.84) - open/close 1644 (time 1763351843.88 total 20.01 last 82.44) - open/close 2495 (time 1763351853.88 total 30.02 last 85.07) - open/close 3328 (time 1763351863.89 total 40.02 last 83.24) - open/close 4158 (time 1763351873.90 total 50.03 last 82.95) - open/close 4971 (time 1763351883.91 total 60.04 last 81.22) - open/close 5782 (time 1763351893.92 total 70.06 last 80.98) - open/close 6667 (time 1763351903.93 total 80.07 last 88.41) - open/close 7529 (time 1763351913.94 total 90.07 last 86.15) - open/close 8450 (time 1763351923.94 total 100.07 last 92.08) - open/close 9298 (time 1763351933.95 total 110.08 last 84.75) total: 10000 open/close in 118.12 seconds: 84.66 ops/second - unlinked 0 (time 1763351944 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 879 (time 1763352026.90 total 10.00 last 87.89) - open/close 1746 (time 1763352036.91 total 20.00 last 86.67) - open/close 2723 (time 1763352046.91 total 30.01 last 97.64) - open/close 3649 (time 1763352056.91 total 40.01 last 92.60) - open/close 4479 (time 1763352066.92 total 50.01 last 82.98) - open/close 5362 (time 1763352076.92 total 60.02 last 88.23) - open/close 6245 (time 1763352086.93 total 70.03 last 88.25) - open/close 7176 (time 1763352096.93 total 80.03 last 93.09) - open/close 8019 (time 1763352106.94 total 90.04 last 84.24) - open/close 8864 (time 1763352116.94 total 100.04 last 84.47) - open/close 9739 (time 1763352126.95 total 110.05 last 87.44) total: 10000 open/close in 112.96 seconds: 88.53 ops/second - unlinked 0 (time 1763352132 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 869 (time 1763352211.45 total 10.01 last 86.80) - open/close 1901 (time 1763352221.46 total 20.02 last 103.10) - open/close 2817 (time 1763352231.47 total 30.03 last 91.52) - open/close 3632 (time 1763352241.48 total 40.04 last 81.41) - open/close 4510 (time 1763352251.49 total 50.04 last 87.76) - open/close 5475 (time 1763352261.49 total 60.05 last 96.44) - open/close 6361 (time 1763352271.50 total 70.05 last 88.59) - open/close 7280 (time 1763352281.51 total 80.06 last 91.80) - open/close 8354 (time 1763352291.51 total 90.06 last 107.40) - open/close 9432 (time 1763352301.51 total 100.06 last 107.79) total: 10000 open/close in 105.30 seconds: 94.97 ops/second - unlinked 0 (time 1763352309 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1309 (time 1763352366.77 total 10.01 last 130.78) - open/close 2385 (time 1763352376.77 total 20.02 last 107.54) - open/close 3335 (time 1763352386.78 total 30.02 last 94.96) - open/close 4399 (time 1763352396.78 total 40.02 last 106.35) - open/close 5686 (time 1763352406.79 total 50.03 last 128.65) - open/close 6616 (time 1763352416.79 total 60.03 last 92.97) - open/close 7617 (time 1763352426.80 total 70.04 last 100.03) - open/close 8667 (time 1763352436.80 total 80.04 last 104.95) - open/close 9675 (time 1763352446.82 total 90.06 last 100.59) total: 10000 open/close in 93.24 seconds: 107.25 ops/second - unlinked 0 (time 1763352452 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1345 (time 1763352512.80 total 10.00 last 134.44) - open/close 2397 (time 1763352522.80 total 20.01 last 105.18) - open/close 3440 (time 1763352532.81 total 30.01 last 104.24) - open/close 4548 (time 1763352542.81 total 40.02 last 110.72) - open/close 5721 (time 1763352552.82 total 50.02 last 117.28) - open/close 6844 (time 1763352562.82 total 60.03 last 112.25) - open/close 7869 (time 1763352572.82 total 70.03 last 102.47) - open/close 9088 (time 1763352582.82 total 80.03 last 121.88) total: 10000 open/close in 88.74 seconds: 112.68 ops/second - unlinked 0 (time 1763352593 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1036 (time 1763352652.69 total 10.01 last 103.52) - open/close 2099 (time 1763352662.69 total 20.01 last 106.28) - open/close 3232 (time 1763352672.69 total 30.01 last 113.27) - open/close 4297 (time 1763352682.70 total 40.02 last 106.45) - open/close 5552 (time 1763352692.71 total 50.03 last 125.40) - open/close 6690 (time 1763352702.71 total 60.03 last 113.79) - open/close 7675 (time 1763352712.71 total 70.03 last 98.47) - open/close 8959 (time 1763352722.71 total 80.03 last 128.37) total: 10000 open/close in 88.72 seconds: 112.71 ops/second - unlinked 0 (time 1763352733 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 945 (time 1763352791.09 total 10.00 last 94.46) - open/close 1965 (time 1763352801.10 total 20.00 last 102.00) - open/close 3225 (time 1763352811.10 total 30.01 last 125.93) - open/close 4437 (time 1763352821.11 total 40.02 last 121.12) - open/close 5670 (time 1763352831.12 total 50.03 last 123.18) - open/close 7105 (time 1763352841.12 total 60.03 last 143.43) - open/close 8426 (time 1763352851.12 total 70.03 last 132.07) - open/close 9611 (time 1763352861.13 total 80.04 last 118.46) total: 10000 open/close in 82.63 seconds: 121.02 ops/second - unlinked 0 (time 1763352865 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1304 (time 1763352921.56 total 10.00 last 130.36) - open/close 2548 (time 1763352931.57 total 20.01 last 124.28) - open/close 4015 (time 1763352941.57 total 30.02 last 146.60) - open/close 5636 (time 1763352951.58 total 40.02 last 162.07) - open/close 6878 (time 1763352961.58 total 50.02 last 124.17) - open/close 7997 (time 1763352971.58 total 60.03 last 111.86) - open/close 9320 (time 1763352981.59 total 70.03 last 132.24) total: 10000 open/close in 76.01 seconds: 131.56 ops/second - unlinked 0 (time 1763352989 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1365 (time 1763353041.10 total 10.00 last 136.49) - open/close 2914 (time 1763353051.10 total 20.01 last 154.82) - open/close 4484 (time 1763353061.11 total 30.01 last 156.90) - open/close 6142 (time 1763353071.11 total 40.01 last 165.77) - open/close 7930 (time 1763353081.11 total 50.02 last 178.79) - open/close 9553 (time 1763353091.12 total 60.02 last 162.20) total: 10000 open/close in 63.51 seconds: 157.46 ops/second - unlinked 0 (time 1763353096 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg437-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2