== conf-sanity test 69: replace an OST with the same index ========================================================== 01:48:03 (1769755683) start mds service on oleg416-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg416-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg416-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 688 (time 1769755798.55 total 10.02 last 68.64) - open/close 1454 (time 1769755808.56 total 20.03 last 76.52) - open/close 2253 (time 1769755818.56 total 30.04 last 79.87) - open/close 3015 (time 1769755828.57 total 40.05 last 76.10) - open/close 3796 (time 1769755838.59 total 50.07 last 77.94) - open/close 4507 (time 1769755848.60 total 60.07 last 71.09) - open/close 5250 (time 1769755858.61 total 70.08 last 74.23) - open/close 6071 (time 1769755868.63 total 80.10 last 81.91) - open/close 6751 (time 1769755878.64 total 90.11 last 67.95) - open/close 7742 (time 1769755888.64 total 100.12 last 99.01) - open/close 8772 (time 1769755898.65 total 110.12 last 102.96) - open/close 9665 (time 1769755908.69 total 120.16 last 88.97) total: 10000 open/close in 124.90 seconds: 80.06 ops/second - unlinked 0 (time 1769755918 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 816 (time 1769756008.21 total 10.01 last 81.54) - open/close 1646 (time 1769756018.22 total 20.01 last 82.95) - open/close 2447 (time 1769756028.23 total 30.02 last 80.03) - open/close 3115 (time 1769756038.25 total 40.04 last 66.68) - open/close 3893 (time 1769756048.26 total 50.06 last 77.66) - open/close 4705 (time 1769756058.27 total 60.06 last 81.18) - open/close 5632 (time 1769756068.27 total 70.07 last 92.63) - open/close 6475 (time 1769756078.28 total 80.07 last 84.27) - open/close 7321 (time 1769756088.28 total 90.07 last 84.59) - open/close 8152 (time 1769756098.30 total 100.09 last 82.96) - open/close 8923 (time 1769756108.30 total 110.09 last 77.08) - open/close 9682 (time 1769756118.31 total 120.10 last 75.85) total: 10000 open/close in 123.56 seconds: 80.93 ops/second - unlinked 0 (time 1769756126 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 737 (time 1769756216.79 total 10.00 last 73.69) - open/close 1552 (time 1769756226.79 total 20.01 last 81.46) - open/close 2569 (time 1769756236.80 total 30.01 last 101.65) - open/close 3424 (time 1769756246.80 total 40.02 last 85.46) - open/close 4191 (time 1769756256.83 total 50.04 last 76.52) - open/close 4947 (time 1769756266.84 total 60.05 last 75.51) - open/close 5514 (time 1769756276.84 total 70.05 last 56.69) - open/close 6259 (time 1769756286.85 total 80.06 last 74.44) - open/close 6896 (time 1769756296.85 total 90.07 last 63.67) - open/close 7685 (time 1769756306.86 total 100.07 last 78.83) - open/close 8445 (time 1769756316.86 total 110.07 last 76.00) - open/close 9255 (time 1769756326.87 total 120.08 last 80.95) - open/close 9930 (time 1769756336.87 total 130.08 last 67.49) total: 10000 open/close in 131.29 seconds: 76.17 ops/second - unlinked 0 (time 1769756343 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 814 (time 1769756439.21 total 10.01 last 81.32) - open/close 1592 (time 1769756449.21 total 20.01 last 77.78) - open/close 2527 (time 1769756459.21 total 30.01 last 93.50) - open/close 3262 (time 1769756469.22 total 40.02 last 73.44) - open/close 4084 (time 1769756479.22 total 50.02 last 82.20) - open/close 4850 (time 1769756489.23 total 60.03 last 76.52) - open/close 5649 (time 1769756499.23 total 70.03 last 79.89) - open/close 6410 (time 1769756509.24 total 80.04 last 76.02) - open/close 7139 (time 1769756519.25 total 90.05 last 72.86) - open/close 7856 (time 1769756529.28 total 100.08 last 71.47) - open/close 8519 (time 1769756539.31 total 110.11 last 66.10) - open/close 9189 (time 1769756549.33 total 120.14 last 66.83) - open/close 9989 (time 1769756559.34 total 130.14 last 79.97) total: 10000 open/close in 130.25 seconds: 76.78 ops/second - unlinked 0 (time 1769756564 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 750 (time 1769756654.68 total 10.00 last 74.98) - open/close 1640 (time 1769756664.70 total 20.02 last 88.86) - open/close 2588 (time 1769756674.71 total 30.03 last 94.70) - open/close 3368 (time 1769756684.71 total 40.03 last 77.99) - open/close 4147 (time 1769756694.71 total 50.03 last 77.88) - open/close 4983 (time 1769756704.71 total 60.04 last 83.56) - open/close 5867 (time 1769756714.72 total 70.04 last 88.33) - open/close 6753 (time 1769756724.73 total 80.05 last 88.50) - open/close 7545 (time 1769756734.74 total 90.06 last 79.18) - open/close 8353 (time 1769756744.76 total 100.08 last 80.63) - open/close 8938 (time 1769756754.79 total 110.12 last 58.28) - open/close 9502 (time 1769756764.82 total 120.14 last 56.28) total: 10000 open/close in 127.90 seconds: 78.19 ops/second - unlinked 0 (time 1769756777 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 772 (time 1769756867.08 total 10.01 last 77.10) - open/close 1532 (time 1769756877.08 total 20.01 last 75.99) - open/close 2241 (time 1769756887.09 total 30.02 last 70.83) - open/close 3025 (time 1769756897.10 total 40.03 last 78.33) - open/close 3803 (time 1769756907.11 total 50.04 last 77.74) - open/close 4594 (time 1769756917.12 total 60.05 last 79.01) - open/close 5383 (time 1769756927.12 total 70.05 last 78.88) - open/close 6068 (time 1769756937.13 total 80.06 last 68.44) - open/close 7054 (time 1769756947.15 total 90.08 last 98.45) - open/close 7734 (time 1769756957.15 total 100.08 last 67.96) - open/close 8492 (time 1769756967.16 total 110.09 last 75.77) - open/close 9549 (time 1769756977.16 total 120.09 last 105.69) total: 10000 open/close in 125.31 seconds: 79.80 ops/second - unlinked 0 (time 1769756987 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 752 (time 1769757071.94 total 10.00 last 75.18) - open/close 1489 (time 1769757081.95 total 20.01 last 73.62) - open/close 2287 (time 1769757091.97 total 30.03 last 79.65) - open/close 2991 (time 1769757101.98 total 40.04 last 70.31) - open/close 3554 (time 1769757111.99 total 50.05 last 56.26) - open/close 4210 (time 1769757122.00 total 60.06 last 65.55) - open/close 4839 (time 1769757132.00 total 70.06 last 62.90) - open/close 5433 (time 1769757142.01 total 80.07 last 59.34) - open/close 6254 (time 1769757152.02 total 90.08 last 82.03) - open/close 7144 (time 1769757162.02 total 100.08 last 88.99) - open/close 8034 (time 1769757172.02 total 110.08 last 88.99) - open/close 8965 (time 1769757182.03 total 120.09 last 93.02) - open/close 9935 (time 1769757192.03 total 130.10 last 96.94) total: 10000 open/close in 130.77 seconds: 76.47 ops/second - unlinked 0 (time 1769757197 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 979 (time 1769757280.50 total 10.02 last 97.70) - open/close 1600 (time 1769757290.51 total 20.03 last 62.05) - open/close 2338 (time 1769757300.53 total 30.05 last 73.64) - open/close 3229 (time 1769757310.54 total 40.05 last 89.08) - open/close 3962 (time 1769757320.54 total 50.06 last 73.28) - open/close 4725 (time 1769757330.54 total 60.06 last 76.27) - open/close 5451 (time 1769757340.54 total 70.06 last 72.59) - open/close 6372 (time 1769757350.55 total 80.07 last 92.04) - open/close 7146 (time 1769757360.55 total 90.07 last 77.40) - open/close 8076 (time 1769757370.56 total 100.07 last 92.94) - open/close 8869 (time 1769757380.57 total 110.08 last 79.23) - open/close 9554 (time 1769757390.57 total 120.09 last 68.46) total: 10000 open/close in 126.20 seconds: 79.24 ops/second - unlinked 0 (time 1769757401 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 764 (time 1769757486.16 total 10.01 last 76.35) - open/close 1601 (time 1769757496.17 total 20.01 last 83.64) - open/close 2333 (time 1769757506.18 total 30.02 last 73.13) - open/close 3200 (time 1769757516.19 total 40.03 last 86.62) - open/close 3950 (time 1769757526.20 total 50.05 last 74.88) - open/close 4778 (time 1769757536.21 total 60.05 last 82.76) - open/close 5670 (time 1769757546.22 total 70.06 last 89.13) - open/close 6545 (time 1769757556.22 total 80.06 last 87.49) - open/close 7414 (time 1769757566.22 total 90.06 last 86.88) - open/close 8299 (time 1769757576.23 total 100.07 last 88.41) - open/close 9206 (time 1769757586.24 total 110.08 last 90.63) total: 10000 open/close in 119.15 seconds: 83.93 ops/second - unlinked 0 (time 1769757599 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1049 (time 1769757680.89 total 10.01 last 104.80) - open/close 1897 (time 1769757690.90 total 20.02 last 84.75) - open/close 2707 (time 1769757700.91 total 30.03 last 80.89) - open/close 3576 (time 1769757710.91 total 40.03 last 86.86) - open/close 4467 (time 1769757720.92 total 50.04 last 89.02) - open/close 5406 (time 1769757730.93 total 60.05 last 93.87) - open/close 6425 (time 1769757740.95 total 70.06 last 101.71) - open/close 7292 (time 1769757750.96 total 80.08 last 86.57) - open/close 7951 (time 1769757760.97 total 90.09 last 65.86) - open/close 8585 (time 1769757770.97 total 100.09 last 63.36) - open/close 9379 (time 1769757780.98 total 110.10 last 79.35) total: 10000 open/close in 117.10 seconds: 85.40 ops/second - unlinked 0 (time 1769757792 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.116@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.116@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.116@tcp sys.timeout=20 start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local oleg416-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg416-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2