== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 07:11:05 (1764936665) Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) oleg425-server: oleg425-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg425-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock oleg425-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 750 (time 1764936775.54 total 10.02 last 74.89) - open/close 1486 (time 1764936785.54 total 20.02 last 73.59) - open/close 2285 (time 1764936795.54 total 30.02 last 79.86) - open/close 3077 (time 1764936805.54 total 40.02 last 79.19) - open/close 3822 (time 1764936815.55 total 50.03 last 74.48) - open/close 4586 (time 1764936825.55 total 60.03 last 76.33) - open/close 5281 (time 1764936835.58 total 70.06 last 69.35) - open/close 5953 (time 1764936845.59 total 80.07 last 67.12) - open/close 6596 (time 1764936855.60 total 90.08 last 64.23) - open/close 7295 (time 1764936865.60 total 100.08 last 69.87) - open/close 7951 (time 1764936875.61 total 110.09 last 65.58) - open/close 8664 (time 1764936885.61 total 120.09 last 71.28) - open/close 9384 (time 1764936895.61 total 130.09 last 71.98) total: 10000 open/close in 138.47 seconds: 72.22 ops/second - unlinked 0 (time 1764936907 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 779 (time 1764936996.71 total 10.00 last 77.90) - open/close 1448 (time 1764937006.71 total 20.00 last 66.89) - open/close 2210 (time 1764937016.72 total 30.01 last 76.17) - open/close 2891 (time 1764937026.73 total 40.01 last 68.05) - open/close 3544 (time 1764937036.73 total 50.02 last 65.26) - open/close 4203 (time 1764937046.73 total 60.02 last 65.90) - open/close 4871 (time 1764937056.75 total 70.03 last 66.71) - open/close 5506 (time 1764937066.75 total 80.04 last 63.45) - open/close 6115 (time 1764937076.76 total 90.05 last 60.86) - open/close 6673 (time 1764937086.76 total 100.05 last 55.79) - open/close 7325 (time 1764937096.78 total 110.06 last 65.11) - open/close 8003 (time 1764937106.78 total 120.07 last 67.79) - open/close 8623 (time 1764937116.78 total 130.07 last 61.97) - open/close 9234 (time 1764937126.79 total 140.07 last 61.09) - open/close 9837 (time 1764937136.80 total 150.08 last 60.24) total: 10000 open/close in 152.51 seconds: 65.57 ops/second - unlinked 0 (time 1764937143 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 714 (time 1764937239.42 total 10.01 last 71.31) - open/close 1440 (time 1764937249.43 total 20.02 last 72.57) - open/close 2150 (time 1764937259.44 total 30.03 last 70.91) - open/close 2829 (time 1764937269.45 total 40.04 last 67.82) - open/close 3510 (time 1764937279.46 total 50.05 last 68.04) - open/close 4254 (time 1764937289.47 total 60.05 last 74.38) - open/close 4995 (time 1764937299.47 total 70.06 last 74.06) - open/close 5700 (time 1764937309.48 total 80.07 last 70.43) - open/close 6441 (time 1764937319.48 total 90.07 last 74.07) - open/close 7122 (time 1764937329.50 total 100.08 last 68.03) - open/close 7825 (time 1764937339.50 total 110.09 last 70.26) - open/close 8576 (time 1764937349.51 total 120.10 last 75.03) - open/close 9220 (time 1764937359.51 total 130.10 last 64.38) - open/close 9889 (time 1764937369.52 total 140.11 last 66.87) total: 10000 open/close in 142.03 seconds: 70.41 ops/second - unlinked 0 (time 1764937375 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 653 (time 1764937481.04 total 10.00 last 65.29) - open/close 1361 (time 1764937491.05 total 20.01 last 70.75) - open/close 2053 (time 1764937501.05 total 30.01 last 69.20) - open/close 2749 (time 1764937511.05 total 40.01 last 69.58) - open/close 3405 (time 1764937521.05 total 50.02 last 65.57) - open/close 4057 (time 1764937531.07 total 60.03 last 65.11) - open/close 4785 (time 1764937541.07 total 70.03 last 72.79) - open/close 5500 (time 1764937551.07 total 80.04 last 71.46) - open/close 6258 (time 1764937561.08 total 90.04 last 75.75) - open/close 7006 (time 1764937571.08 total 100.05 last 74.78) - open/close 7704 (time 1764937581.10 total 110.06 last 69.68) - open/close 8419 (time 1764937591.11 total 120.07 last 71.45) - open/close 9135 (time 1764937601.12 total 130.08 last 71.54) - open/close 9854 (time 1764937611.12 total 140.08 last 71.88) total: 10000 open/close in 142.05 seconds: 70.40 ops/second - unlinked 0 (time 1764937616 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 721 (time 1764937713.78 total 10.01 last 72.02) - open/close 1495 (time 1764937723.79 total 20.02 last 77.36) - open/close 2246 (time 1764937733.81 total 30.04 last 74.92) - open/close 2947 (time 1764937743.82 total 40.05 last 70.05) - open/close 3681 (time 1764937753.83 total 50.05 last 73.34) - open/close 4433 (time 1764937763.84 total 60.06 last 75.13) - open/close 5206 (time 1764937773.84 total 70.07 last 77.29) - open/close 5887 (time 1764937783.84 total 80.07 last 68.09) - open/close 6562 (time 1764937793.84 total 90.07 last 67.48) - open/close 7273 (time 1764937803.86 total 100.08 last 71.00) - open/close 8048 (time 1764937813.86 total 110.09 last 77.48) - open/close 8763 (time 1764937823.86 total 120.09 last 71.48) - open/close 9409 (time 1764937833.87 total 130.09 last 64.57) total: 10000 open/close in 138.90 seconds: 71.99 ops/second - unlinked 0 (time 1764937846 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 685 (time 1764937942.31 total 10.01 last 68.44) - open/close 1379 (time 1764937952.31 total 20.02 last 69.34) - open/close 2111 (time 1764937962.33 total 30.03 last 73.10) - open/close 2855 (time 1764937972.34 total 40.04 last 74.34) - open/close 3591 (time 1764937982.34 total 50.05 last 73.54) - open/close 4304 (time 1764937992.35 total 60.06 last 71.23) - open/close 5047 (time 1764938002.36 total 70.06 last 74.29) - open/close 5740 (time 1764938012.37 total 80.08 last 69.18) - open/close 6408 (time 1764938022.38 total 90.08 last 66.77) - open/close 7129 (time 1764938032.39 total 100.09 last 72.02) - open/close 7895 (time 1764938042.39 total 110.10 last 76.56) - open/close 8596 (time 1764938052.40 total 120.10 last 70.06) - open/close 9294 (time 1764938062.41 total 130.11 last 69.75) total: 10000 open/close in 139.63 seconds: 71.62 ops/second - unlinked 0 (time 1764938075 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 763 (time 1764938167.88 total 10.02 last 76.17) - open/close 1494 (time 1764938177.89 total 20.02 last 73.06) - open/close 2226 (time 1764938187.89 total 30.03 last 73.18) - open/close 3024 (time 1764938197.90 total 40.03 last 79.76) - open/close 3738 (time 1764938207.91 total 50.05 last 71.27) - open/close 4432 (time 1764938217.92 total 60.06 last 69.33) - open/close 5154 (time 1764938227.93 total 70.06 last 72.18) - open/close 5935 (time 1764938237.93 total 80.07 last 78.04) - open/close 6683 (time 1764938247.94 total 90.07 last 74.77) - open/close 7369 (time 1764938257.95 total 100.08 last 68.53) - open/close 8097 (time 1764938267.95 total 110.09 last 72.79) - open/close 8829 (time 1764938277.96 total 120.09 last 73.16) - open/close 9515 (time 1764938287.96 total 130.10 last 68.57) total: 10000 open/close in 137.29 seconds: 72.84 ops/second - unlinked 0 (time 1764938298 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 767 (time 1764938394.44 total 10.01 last 76.64) - open/close 1486 (time 1764938404.45 total 20.02 last 71.79) - open/close 2152 (time 1764938414.45 total 30.03 last 66.58) - open/close 2746 (time 1764938424.46 total 40.03 last 59.35) - open/close 3315 (time 1764938434.47 total 50.04 last 56.84) - open/close 3958 (time 1764938444.48 total 60.05 last 64.24) - open/close 4605 (time 1764938454.48 total 70.06 last 64.69) - open/close 5285 (time 1764938464.49 total 80.06 last 68.00) - open/close 5993 (time 1764938474.49 total 90.06 last 70.75) - open/close 6776 (time 1764938484.50 total 100.07 last 78.26) - open/close 7500 (time 1764938494.50 total 110.07 last 72.36) - open/close 8204 (time 1764938504.50 total 120.07 last 70.40) - open/close 8906 (time 1764938514.50 total 130.07 last 70.19) - open/close 9648 (time 1764938524.51 total 140.08 last 74.17) total: 10000 open/close in 144.86 seconds: 69.03 ops/second - unlinked 0 (time 1764938532 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 755 (time 1764938623.90 total 10.01 last 75.40) - open/close 1510 (time 1764938633.90 total 20.02 last 75.45) - open/close 2339 (time 1764938643.90 total 30.02 last 82.89) - open/close 3098 (time 1764938653.91 total 40.03 last 75.85) - open/close 3806 (time 1764938663.92 total 50.03 last 70.75) - open/close 4443 (time 1764938673.93 total 60.04 last 63.64) - open/close 5173 (time 1764938683.94 total 70.05 last 72.94) - open/close 5910 (time 1764938693.94 total 80.06 last 73.67) - open/close 6688 (time 1764938703.96 total 90.07 last 77.67) - open/close 7390 (time 1764938713.97 total 100.09 last 70.10) - open/close 8103 (time 1764938723.98 total 110.09 last 71.26) - open/close 8831 (time 1764938733.99 total 120.11 last 72.72) - open/close 9553 (time 1764938743.99 total 130.11 last 72.17) total: 10000 open/close in 136.96 seconds: 73.01 ops/second - unlinked 0 (time 1764938755 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 760 (time 1764938851.41 total 10.00 last 75.96) - open/close 1473 (time 1764938861.42 total 20.02 last 71.21) - open/close 2196 (time 1764938871.43 total 30.03 last 72.24) - open/close 2914 (time 1764938881.43 total 40.03 last 71.79) - open/close 3682 (time 1764938891.44 total 50.04 last 76.72) - open/close 4405 (time 1764938901.44 total 60.04 last 72.28) - open/close 5111 (time 1764938911.44 total 70.04 last 70.59) - open/close 5829 (time 1764938921.45 total 80.05 last 71.72) - open/close 6560 (time 1764938931.46 total 90.06 last 73.05) - open/close 7288 (time 1764938941.47 total 100.06 last 72.77) - open/close 7987 (time 1764938951.47 total 110.07 last 69.88) - open/close 8694 (time 1764938961.48 total 120.07 last 70.65) - open/close 9401 (time 1764938971.48 total 130.08 last 70.68) total: 10000 open/close in 138.04 seconds: 72.44 ops/second - unlinked 0 (time 1764938983 ; total 1 ; last 1) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 765 (time 1764939075.14 total 10.01 last 76.43) - open/close 1567 (time 1764939085.14 total 20.01 last 80.19) - open/close 2368 (time 1764939095.15 total 30.02 last 80.00) - open/close 3107 (time 1764939105.15 total 40.02 last 73.89) - open/close 3903 (time 1764939115.16 total 50.03 last 79.53) - open/close 4726 (time 1764939125.17 total 60.04 last 82.26) - open/close 5458 (time 1764939135.17 total 70.04 last 73.18) - open/close 6179 (time 1764939145.18 total 80.04 last 72.06) - open/close 6880 (time 1764939155.18 total 90.05 last 70.07) - open/close 7609 (time 1764939165.19 total 100.06 last 72.79) - open/close 8369 (time 1764939175.20 total 110.07 last 75.92) - open/close 9121 (time 1764939185.21 total 120.08 last 75.18) - open/close 9916 (time 1764939195.21 total 130.08 last 79.47) total: 10000 open/close in 131.20 seconds: 76.22 ops/second - unlinked 0 (time 1764939199 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5120 376448 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35145 s, 3.0 MB/s fail_loc=0x80000204 Failing ost1 on oleg425-server Stopping /mnt/lustre-ost1 (opts:) on oleg425-server reboot facets: ost1 Failover ost1 to oleg425-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1764939360.71 total 35.93 last 0.00) total: 100 open/close in 37.94 seconds: 2.64 ops/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg425-server: LNET ready to unload oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.