== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:12:55 (1781280775) Stopping clients: oleg337-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg337-client.virtnet /mnt/lustre2 (opts:-f) oleg337-server: oleg337-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg337-server: detected 4 online CPUs by sysfs oleg337-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg337-server: detected 4 online CPUs by sysfs oleg337-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg337-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg337-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 839 (time 1781280915.15 total 10.01 last 83.84) - open/close 1780 (time 1781280925.15 total 20.01 last 94.06) - open/close 2588 (time 1781280935.15 total 30.01 last 80.78) - open/close 3439 (time 1781280945.16 total 40.03 last 85.00) - open/close 4243 (time 1781280955.17 total 50.03 last 80.37) - open/close 5031 (time 1781280965.18 total 60.04 last 78.73) - open/close 5755 (time 1781280975.18 total 70.04 last 72.37) - open/close 6306 (time 1781280985.18 total 80.04 last 55.08) - open/close 6915 (time 1781280995.19 total 90.05 last 60.89) - open/close 7601 (time 1781281005.20 total 100.06 last 68.51) - open/close 8343 (time 1781281015.22 total 110.08 last 74.02) - open/close 8991 (time 1781281025.23 total 120.09 last 64.78) - open/close 9759 (time 1781281035.23 total 130.09 last 76.75) total: 10000 open/close in 132.66 seconds: 75.38 ops/second - unlinked 0 (time 1781281042 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 736 (time 1781281133.06 total 10.02 last 73.43) - open/close 1559 (time 1781281143.07 total 20.02 last 82.28) - open/close 2411 (time 1781281153.08 total 30.04 last 85.09) - open/close 3231 (time 1781281163.09 total 40.05 last 81.91) - open/close 4094 (time 1781281173.09 total 50.05 last 86.25) - open/close 4929 (time 1781281183.12 total 60.08 last 83.32) - open/close 5756 (time 1781281193.13 total 70.08 last 82.62) - open/close 6562 (time 1781281203.13 total 80.09 last 80.60) - open/close 7406 (time 1781281213.13 total 90.09 last 84.32) - open/close 8210 (time 1781281223.14 total 100.10 last 80.39) - open/close 9029 (time 1781281233.14 total 110.10 last 81.90) - open/close 9829 (time 1781281243.14 total 120.10 last 79.96) total: 10000 open/close in 122.29 seconds: 81.78 ops/second - unlinked 0 (time 1781281249 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 804 (time 1781281338.51 total 10.01 last 80.34) - open/close 1653 (time 1781281348.52 total 20.02 last 84.77) - open/close 2450 (time 1781281358.54 total 30.04 last 79.58) - open/close 3264 (time 1781281368.55 total 40.05 last 81.34) - open/close 4092 (time 1781281378.55 total 50.05 last 82.78) - open/close 4854 (time 1781281388.56 total 60.06 last 76.13) - open/close 5643 (time 1781281398.57 total 70.07 last 78.84) - open/close 6299 (time 1781281408.58 total 80.08 last 65.53) - open/close 6986 (time 1781281418.58 total 90.08 last 68.67) - open/close 7706 (time 1781281428.58 total 100.08 last 71.98) - open/close 8424 (time 1781281438.60 total 110.10 last 71.67) - open/close 9224 (time 1781281448.61 total 120.11 last 79.92) total: 10000 open/close in 129.90 seconds: 76.98 ops/second - unlinked 0 (time 1781281462 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 828 (time 1781281549.43 total 10.01 last 82.71) - open/close 1359 (time 1781281559.44 total 20.02 last 53.06) - open/close 2249 (time 1781281569.44 total 30.02 last 89.00) - open/close 3127 (time 1781281579.44 total 40.03 last 87.75) - open/close 3984 (time 1781281589.45 total 50.03 last 85.65) - open/close 4827 (time 1781281599.47 total 60.05 last 84.14) - open/close 5649 (time 1781281609.48 total 70.06 last 82.14) - open/close 6403 (time 1781281619.48 total 80.06 last 75.38) - open/close 7189 (time 1781281629.48 total 90.06 last 78.57) - open/close 7868 (time 1781281639.49 total 100.07 last 67.86) - open/close 8791 (time 1781281649.49 total 110.07 last 92.30) - open/close 9562 (time 1781281659.50 total 120.08 last 77.00) total: 10000 open/close in 125.67 seconds: 79.57 ops/second - unlinked 0 (time 1781281669 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 885 (time 1781281757.56 total 10.00 last 88.48) - open/close 1767 (time 1781281767.56 total 20.01 last 88.16) - open/close 2558 (time 1781281777.57 total 30.02 last 79.01) - open/close 3425 (time 1781281787.58 total 40.02 last 86.63) - open/close 4216 (time 1781281797.58 total 50.03 last 79.08) - open/close 4980 (time 1781281807.59 total 60.04 last 76.34) - open/close 5879 (time 1781281817.60 total 70.05 last 89.78) - open/close 6681 (time 1781281827.61 total 80.06 last 80.11) - open/close 7503 (time 1781281837.63 total 90.08 last 82.08) - open/close 8309 (time 1781281847.63 total 100.08 last 80.58) - open/close 9119 (time 1781281857.64 total 110.08 last 80.96) - open/close 9968 (time 1781281867.64 total 120.08 last 84.89) total: 10000 open/close in 120.50 seconds: 82.99 ops/second - unlinked 0 (time 1781281871 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 824 (time 1781281953.75 total 10.00 last 82.37) - open/close 1618 (time 1781281963.76 total 20.01 last 79.34) - open/close 2527 (time 1781281973.76 total 30.02 last 90.82) - open/close 3357 (time 1781281983.77 total 40.02 last 82.98) - open/close 4196 (time 1781281993.77 total 50.02 last 83.89) - open/close 5091 (time 1781282003.79 total 60.05 last 89.29) - open/close 6053 (time 1781282013.80 total 70.05 last 96.14) - open/close 7035 (time 1781282023.80 total 80.05 last 98.17) - open/close 7859 (time 1781282033.81 total 90.06 last 82.34) - open/close 8716 (time 1781282043.81 total 100.07 last 85.67) - open/close 9522 (time 1781282053.82 total 110.07 last 80.54) total: 10000 open/close in 115.42 seconds: 86.64 ops/second - unlinked 0 (time 1781282063 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 840 (time 1781282150.20 total 10.00 last 83.98) - open/close 1695 (time 1781282160.22 total 20.02 last 85.36) - open/close 2592 (time 1781282170.22 total 30.02 last 89.65) - open/close 3424 (time 1781282180.23 total 40.03 last 83.18) - open/close 4268 (time 1781282190.24 total 50.04 last 84.30) - open/close 5088 (time 1781282200.25 total 60.05 last 81.91) - open/close 5891 (time 1781282210.26 total 70.06 last 80.21) - open/close 6719 (time 1781282220.26 total 80.06 last 82.78) - open/close 7559 (time 1781282230.27 total 90.07 last 83.92) - open/close 8431 (time 1781282240.28 total 100.08 last 87.17) - open/close 9256 (time 1781282250.28 total 110.08 last 82.43) total: 10000 open/close in 118.81 seconds: 84.17 ops/second - unlinked 0 (time 1781282263 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 831 (time 1781282348.35 total 10.02 last 82.93) - open/close 1632 (time 1781282358.36 total 20.03 last 80.05) - open/close 2543 (time 1781282368.36 total 30.03 last 91.10) - open/close 3428 (time 1781282378.38 total 40.04 last 88.35) - open/close 4263 (time 1781282388.39 total 50.05 last 83.42) - open/close 5152 (time 1781282398.40 total 60.06 last 88.83) - open/close 5978 (time 1781282408.40 total 70.07 last 82.55) - open/close 6848 (time 1781282418.41 total 80.07 last 86.97) - open/close 7683 (time 1781282428.41 total 90.07 last 83.49) - open/close 8563 (time 1781282438.41 total 100.08 last 87.96) - open/close 9381 (time 1781282448.42 total 110.09 last 81.74) total: 10000 open/close in 117.52 seconds: 85.09 ops/second - unlinked 0 (time 1781282460 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 883 (time 1781282547.96 total 10.00 last 88.27) - open/close 1738 (time 1781282557.97 total 20.02 last 85.36) - open/close 2573 (time 1781282567.98 total 30.03 last 83.45) - open/close 3381 (time 1781282577.99 total 40.03 last 80.75) - open/close 4283 (time 1781282588.00 total 50.04 last 90.11) - open/close 5113 (time 1781282598.00 total 60.05 last 82.93) - open/close 5944 (time 1781282608.02 total 70.06 last 83.00) - open/close 6739 (time 1781282618.03 total 80.08 last 79.37) - open/close 7521 (time 1781282628.04 total 90.09 last 78.13) - open/close 8406 (time 1781282638.05 total 100.09 last 88.47) - open/close 9304 (time 1781282648.05 total 110.10 last 89.73) total: 10000 open/close in 118.49 seconds: 84.39 ops/second - unlinked 0 (time 1781282659 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 844 (time 1781282742.83 total 10.01 last 84.35) - open/close 1702 (time 1781282752.85 total 20.02 last 85.67) - open/close 2486 (time 1781282762.85 total 30.02 last 78.38) - open/close 3313 (time 1781282772.85 total 40.02 last 82.70) - open/close 4104 (time 1781282782.85 total 50.03 last 79.09) - open/close 4965 (time 1781282792.85 total 60.03 last 86.08) - open/close 5775 (time 1781282802.87 total 70.04 last 80.88) - open/close 6620 (time 1781282812.89 total 80.06 last 84.34) - open/close 7444 (time 1781282822.90 total 90.07 last 82.33) - open/close 8292 (time 1781282832.90 total 100.08 last 84.74) - open/close 9118 (time 1781282842.91 total 110.09 last 82.50) - open/close 9949 (time 1781282852.92 total 120.09 last 83.06) total: 10000 open/close in 120.51 seconds: 82.98 ops/second - unlinked 0 (time 1781282858 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 850 (time 1781282944.07 total 10.01 last 84.90) - open/close 1676 (time 1781282954.08 total 20.02 last 82.56) - open/close 2557 (time 1781282964.08 total 30.02 last 88.06) - open/close 3434 (time 1781282974.09 total 40.03 last 87.66) - open/close 4268 (time 1781282984.09 total 50.03 last 83.38) - open/close 5104 (time 1781282994.10 total 60.04 last 83.49) - open/close 5975 (time 1781283004.10 total 70.04 last 87.07) - open/close 6724 (time 1781283014.12 total 80.06 last 74.79) - open/close 7528 (time 1781283024.12 total 90.06 last 80.39) - open/close 8381 (time 1781283034.12 total 100.06 last 85.29) - open/close 9169 (time 1781283044.14 total 110.08 last 78.66) - open/close 9973 (time 1781283054.17 total 120.11 last 80.12) total: 10000 open/close in 120.47 seconds: 83.01 ops/second - unlinked 0 (time 1781283058 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5248 376320 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208874 s, 5.0 MB/s fail_loc=0x80000204 Failing ost1 on oleg337-server Stopping /mnt/lustre-ost1 (opts:) on oleg337-server reboot facets: ost1 Failover ost1 to oleg337-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.53 seconds: 39.47 ops/second umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg337-server: LNET ready to unload oleg337-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.