== conf-sanity test 69: replace an OST with the same index ========================================================== 06:40:40 (1756809640) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1211 (time 1756809688.92 total 10.00 last 121.05) - open/close 2561 (time 1756809698.93 total 20.01 last 134.92) - open/close 3845 (time 1756809708.94 total 30.02 last 128.31) - open/close 4966 (time 1756809718.94 total 40.02 last 112.04) - open/close 6131 (time 1756809728.94 total 50.02 last 116.47) - open/close 7240 (time 1756809738.95 total 60.03 last 110.78) - open/close 8572 (time 1756809748.96 total 70.04 last 133.13) - open/close 9910 (time 1756809758.96 total 80.04 last 133.78) total: 10000 open/close in 80.82 seconds: 123.73 ops/second - unlinked 0 (time 1756809762 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1741 (time 1756809812.61 total 10.00 last 174.02) - open/close 3596 (time 1756809822.61 total 20.01 last 185.45) - open/close 5307 (time 1756809832.61 total 30.01 last 171.08) - open/close 7132 (time 1756809842.61 total 40.01 last 182.41) - open/close 8924 (time 1756809852.62 total 50.02 last 179.15) total: 10000 open/close in 55.39 seconds: 180.54 ops/second - unlinked 0 (time 1756809859 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1916 (time 1756809898.08 total 10.00 last 191.60) - open/close 3470 (time 1756809908.09 total 20.00 last 155.34) - open/close 5259 (time 1756809918.09 total 30.00 last 178.90) - open/close 7518 (time 1756809928.09 total 40.01 last 225.81) - open/close 9401 (time 1756809938.09 total 50.01 last 188.30) total: 10000 open/close in 52.18 seconds: 191.63 ops/second - unlinked 0 (time 1756809942 ; total 1 ; last 1) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1833 (time 1756809982.50 total 10.00 last 183.27) - open/close 4069 (time 1756809992.50 total 20.01 last 223.52) - open/close 6791 (time 1756810002.50 total 30.01 last 272.18) - open/close 8983 (time 1756810012.50 total 40.01 last 219.15) total: 10000 open/close in 43.47 seconds: 230.03 ops/second - unlinked 0 (time 1756810017 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 1161 (time 1756810049.16 total 10.01 last 115.97) - open/close 2232 (time 1756810059.19 total 20.04 last 106.82) - open/close 3344 (time 1756810069.19 total 30.04 last 111.16) - open/close 5153 (time 1756810079.20 total 40.05 last 180.82) - open/close 6993 (time 1756810089.20 total 50.05 last 183.94) - open/close 8599 (time 1756810099.20 total 60.05 last 160.55) total: 10000 open/close in 67.49 seconds: 148.17 ops/second - unlinked 0 (time 1756810108 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1771 (time 1756810157.47 total 10.00 last 177.04) - open/close 3727 (time 1756810167.48 total 20.01 last 195.46) - open/close 5556 (time 1756810177.48 total 30.01 last 182.87) - open/close 7588 (time 1756810187.48 total 40.01 last 203.19) - open/close 8923 (time 1756810197.48 total 50.01 last 133.49) total: 10000 open/close in 59.28 seconds: 168.70 ops/second - unlinked 0 (time 1756810208 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1278 (time 1756810266.33 total 10.00 last 127.79) - open/close 2568 (time 1756810276.34 total 20.00 last 128.95) - open/close 3816 (time 1756810286.34 total 30.01 last 124.74) - open/close 5458 (time 1756810296.35 total 40.02 last 164.09) - open/close 7348 (time 1756810306.36 total 50.02 last 188.85) - open/close 9622 (time 1756810316.36 total 60.02 last 227.39) total: 10000 open/close in 61.34 seconds: 163.03 ops/second - unlinked 0 (time 1756810319 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2127 (time 1756810354.16 total 10.01 last 212.45) - open/close 3834 (time 1756810364.16 total 20.01 last 170.68) - open/close 5460 (time 1756810374.16 total 30.02 last 162.54) - open/close 7095 (time 1756810384.16 total 40.02 last 163.49) - open/close 8949 (time 1756810394.17 total 50.02 last 185.36) total: 10000 open/close in 54.63 seconds: 183.05 ops/second - unlinked 0 (time 1756810400 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2025 (time 1756810438.30 total 10.00 last 202.47) - open/close 4100 (time 1756810448.31 total 20.00 last 207.48) - open/close 6080 (time 1756810458.31 total 30.00 last 197.99) - open/close 7738 (time 1756810468.31 total 40.01 last 165.74) - open/close 9696 (time 1756810478.31 total 50.01 last 195.79) total: 10000 open/close in 51.11 seconds: 195.65 ops/second - unlinked 0 (time 1756810480 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1867 (time 1756810519.95 total 10.00 last 186.63) - open/close 3659 (time 1756810529.96 total 20.01 last 179.13) - open/close 5519 (time 1756810539.96 total 30.01 last 185.98) - open/close 7507 (time 1756810549.96 total 40.01 last 198.78) - open/close 9597 (time 1756810559.96 total 50.01 last 208.98) total: 10000 open/close in 51.83 seconds: 192.93 ops/second - unlinked 0 (time 1756810563 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre On OST0, 17269 used inodes rc=0 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.133@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.133@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.133@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.204.133@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.204.133@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2