== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 12:17:24 (1781281044) Stopping clients: oleg622-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg622-client.virtnet /mnt/lustre2 (opts:-f) oleg622-server: oleg622-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing load_modules_local oleg622-server: Loading modules from /home/green/git/lustre-release/lustre oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg622-server: detected 4 online CPUs by sysfs oleg622-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing load_modules_local oleg622-server: Loading modules from /home/green/git/lustre-release/lustre oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg622-server: detected 4 online CPUs by sysfs oleg622-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg622-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg622-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg622-client.virtnet: -o user_xattr,flock oleg622-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 577 (time 1781281180.65 total 10.01 last 57.67) - open/close 1190 (time 1781281190.65 total 20.01 last 61.30) - open/close 1817 (time 1781281200.65 total 30.01 last 62.68) - open/close 2448 (time 1781281210.66 total 40.01 last 63.08) - open/close 3085 (time 1781281220.67 total 50.03 last 63.61) - open/close 3710 (time 1781281230.67 total 60.03 last 62.50) - open/close 4334 (time 1781281240.69 total 70.04 last 62.29) - open/close 4985 (time 1781281250.71 total 80.06 last 64.99) - open/close 5637 (time 1781281260.71 total 90.07 last 65.15) - open/close 6248 (time 1781281270.73 total 100.08 last 61.01) - open/close 6885 (time 1781281280.73 total 110.08 last 63.69) - open/close 7498 (time 1781281290.73 total 120.09 last 61.26) - open/close 8138 (time 1781281300.76 total 130.12 last 63.83) - open/close 8768 (time 1781281310.78 total 140.13 last 62.90) - open/close 9360 (time 1781281320.79 total 150.14 last 59.14) - open/close 9954 (time 1781281330.80 total 160.15 last 59.34) total: 10000 open/close in 160.79 seconds: 62.19 ops/second - unlinked 0 (time 1781281335 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 644 (time 1781281440.65 total 10.01 last 64.32) - open/close 1296 (time 1781281450.66 total 20.02 last 65.15) - open/close 1950 (time 1781281460.67 total 30.03 last 65.33) - open/close 2604 (time 1781281470.68 total 40.04 last 65.35) - open/close 3289 (time 1781281480.68 total 50.04 last 68.47) - open/close 3951 (time 1781281490.69 total 60.05 last 66.14) - open/close 4593 (time 1781281500.70 total 70.06 last 64.17) - open/close 5257 (time 1781281510.71 total 80.07 last 66.30) - open/close 5847 (time 1781281520.75 total 90.11 last 58.77) - open/close 6504 (time 1781281530.77 total 100.13 last 65.59) - open/close 7230 (time 1781281540.77 total 110.13 last 72.60) - open/close 7941 (time 1781281550.78 total 120.14 last 71.05) - open/close 8655 (time 1781281560.79 total 130.14 last 71.34) - open/close 9343 (time 1781281570.79 total 140.15 last 68.77) total: 10000 open/close in 148.71 seconds: 67.25 ops/second - unlinked 0 (time 1781281583 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 700 (time 1781281685.32 total 10.01 last 69.92) - open/close 1418 (time 1781281695.33 total 20.02 last 71.74) - open/close 2127 (time 1781281705.34 total 30.03 last 70.83) - open/close 2805 (time 1781281715.35 total 40.03 last 67.76) - open/close 3532 (time 1781281725.35 total 50.03 last 72.70) - open/close 4282 (time 1781281735.35 total 60.04 last 74.96) - open/close 4953 (time 1781281745.36 total 70.05 last 67.05) - open/close 5607 (time 1781281755.36 total 80.05 last 65.40) - open/close 6244 (time 1781281765.36 total 90.05 last 63.68) - open/close 6876 (time 1781281775.37 total 100.06 last 63.14) - open/close 7589 (time 1781281785.38 total 110.07 last 71.26) - open/close 8319 (time 1781281795.38 total 120.07 last 72.98) - open/close 9009 (time 1781281805.38 total 130.07 last 68.98) - open/close 9725 (time 1781281815.39 total 140.08 last 71.55) total: 10000 open/close in 144.08 seconds: 69.40 ops/second - unlinked 0 (time 1781281822 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 898 (time 1781281911.38 total 10.01 last 89.73) - open/close 1784 (time 1781281921.38 total 20.01 last 88.56) - open/close 2527 (time 1781281931.40 total 30.03 last 74.20) - open/close 3371 (time 1781281941.41 total 40.04 last 84.25) - open/close 4083 (time 1781281951.43 total 50.06 last 71.09) - open/close 4921 (time 1781281961.44 total 60.07 last 83.68) - open/close 5724 (time 1781281971.45 total 70.08 last 80.22) - open/close 6468 (time 1781281981.46 total 80.09 last 74.33) - open/close 7353 (time 1781281991.47 total 90.10 last 88.40) - open/close 8193 (time 1781282001.49 total 100.11 last 83.91) - open/close 8974 (time 1781282011.49 total 110.12 last 78.07) - open/close 9723 (time 1781282021.50 total 120.13 last 74.83) total: 10000 open/close in 123.81 seconds: 80.77 ops/second - unlinked 0 (time 1781282028 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 908 (time 1781282112.47 total 10.01 last 90.73) - open/close 1864 (time 1781282122.48 total 20.01 last 95.53) - open/close 2572 (time 1781282132.48 total 30.02 last 70.79) - open/close 3286 (time 1781282142.48 total 40.02 last 71.39) - open/close 4186 (time 1781282152.48 total 50.02 last 89.96) - open/close 4983 (time 1781282162.49 total 60.03 last 79.64) - open/close 5815 (time 1781282172.51 total 70.04 last 83.06) - open/close 6502 (time 1781282182.51 total 80.05 last 68.65) - open/close 7192 (time 1781282192.53 total 90.06 last 68.91) - open/close 7841 (time 1781282202.54 total 100.08 last 64.81) - open/close 8699 (time 1781282212.55 total 110.09 last 85.70) - open/close 9521 (time 1781282222.56 total 120.10 last 82.12) total: 10000 open/close in 126.51 seconds: 79.04 ops/second - unlinked 0 (time 1781282231 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 720 (time 1781282317.28 total 10.01 last 71.89) - open/close 1393 (time 1781282327.29 total 20.02 last 67.25) - open/close 2075 (time 1781282337.29 total 30.03 last 68.18) - open/close 2752 (time 1781282347.30 total 40.04 last 67.60) - open/close 3389 (time 1781282357.31 total 50.05 last 63.67) - open/close 4109 (time 1781282367.31 total 60.05 last 71.97) - open/close 4837 (time 1781282377.33 total 70.06 last 72.69) - open/close 5554 (time 1781282387.34 total 80.07 last 71.65) - open/close 6335 (time 1781282397.34 total 90.07 last 78.09) - open/close 7190 (time 1781282407.34 total 100.08 last 85.48) - open/close 7893 (time 1781282417.35 total 110.08 last 70.26) - open/close 8625 (time 1781282427.36 total 120.09 last 73.13) - open/close 9271 (time 1781282437.36 total 130.10 last 64.56) - open/close 9906 (time 1781282447.38 total 140.11 last 63.39) total: 10000 open/close in 141.43 seconds: 70.71 ops/second - unlinked 0 (time 1781282452 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 682 (time 1781282553.65 total 10.00 last 68.18) - open/close 1360 (time 1781282563.65 total 20.00 last 67.79) - open/close 1890 (time 1781282573.66 total 30.02 last 52.94) - open/close 2552 (time 1781282583.66 total 40.02 last 66.16) - open/close 3211 (time 1781282593.68 total 50.04 last 65.79) - open/close 3848 (time 1781282603.69 total 60.04 last 63.66) - open/close 4562 (time 1781282613.69 total 70.05 last 71.38) - open/close 5253 (time 1781282623.70 total 80.05 last 69.05) - open/close 5943 (time 1781282633.70 total 90.06 last 68.96) - open/close 6547 (time 1781282643.71 total 100.06 last 60.38) - open/close 7224 (time 1781282653.71 total 110.06 last 67.69) - open/close 7865 (time 1781282663.71 total 120.07 last 64.06) - open/close 8540 (time 1781282673.71 total 130.07 last 67.49) - open/close 9226 (time 1781282683.72 total 140.08 last 68.55) - open/close 9931 (time 1781282693.73 total 150.09 last 70.42) total: 10000 open/close in 151.21 seconds: 66.13 ops/second - unlinked 0 (time 1781282699 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 835 (time 1781282795.66 total 10.01 last 83.41) - open/close 1557 (time 1781282805.67 total 20.02 last 72.13) - open/close 2276 (time 1781282815.67 total 30.02 last 71.89) - open/close 2915 (time 1781282825.70 total 40.05 last 63.72) - open/close 3571 (time 1781282835.71 total 50.06 last 65.53) - open/close 4264 (time 1781282845.72 total 60.07 last 69.25) - open/close 4895 (time 1781282855.73 total 70.08 last 63.05) - open/close 5531 (time 1781282865.74 total 80.09 last 63.53) - open/close 6218 (time 1781282875.74 total 90.09 last 68.69) - open/close 6943 (time 1781282885.75 total 100.10 last 72.44) - open/close 7598 (time 1781282895.75 total 110.10 last 65.50) - open/close 8315 (time 1781282905.76 total 120.10 last 71.66) - open/close 9032 (time 1781282915.77 total 130.12 last 71.59) - open/close 9708 (time 1781282925.79 total 140.14 last 67.47) total: 10000 open/close in 144.70 seconds: 69.11 ops/second - unlinked 0 (time 1781282934 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 713 (time 1781283021.57 total 10.00 last 71.27) - open/close 1394 (time 1781283031.57 total 20.00 last 68.10) - open/close 2090 (time 1781283041.57 total 30.00 last 69.60) - open/close 2837 (time 1781283051.58 total 40.01 last 74.67) - open/close 3593 (time 1781283061.59 total 50.02 last 75.54) - open/close 4268 (time 1781283071.60 total 60.03 last 67.43) - open/close 5008 (time 1781283081.60 total 70.04 last 73.93) - open/close 5745 (time 1781283091.62 total 80.05 last 73.58) - open/close 6571 (time 1781283101.63 total 90.06 last 82.50) - open/close 7464 (time 1781283111.64 total 100.07 last 89.22) - open/close 8290 (time 1781283121.65 total 110.08 last 82.56) - open/close 9049 (time 1781283131.65 total 120.08 last 75.87) - open/close 9763 (time 1781283141.66 total 130.09 last 71.35) total: 10000 open/close in 133.35 seconds: 74.99 ops/second - unlinked 0 (time 1781283148 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 776 (time 1781283243.95 total 10.01 last 77.51) - open/close 1508 (time 1781283253.96 total 20.01 last 73.19) - open/close 2281 (time 1781283263.97 total 30.03 last 77.19) - open/close 2970 (time 1781283273.97 total 40.03 last 68.90) - open/close 3610 (time 1781283283.99 total 50.04 last 63.89) - open/close 4344 (time 1781283294.01 total 60.07 last 73.24) - open/close 5097 (time 1781283304.01 total 70.07 last 75.28) - open/close 5842 (time 1781283314.02 total 80.07 last 74.47) - open/close 6532 (time 1781283324.03 total 90.09 last 68.89) - open/close 7299 (time 1781283334.03 total 100.09 last 76.68) - open/close 7937 (time 1781283344.05 total 110.11 last 63.71) - open/close 8624 (time 1781283354.05 total 120.11 last 68.66) - open/close 9293 (time 1781283364.06 total 130.12 last 66.86) - open/close 9936 (time 1781283374.07 total 140.12 last 64.26) total: 10000 open/close in 141.05 seconds: 70.90 ops/second - unlinked 0 (time 1781283378 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 734 (time 1781283470.58 total 10.00 last 73.39) - open/close 1501 (time 1781283480.59 total 20.01 last 76.62) - open/close 2217 (time 1781283490.60 total 30.02 last 71.56) - open/close 2914 (time 1781283500.61 total 40.03 last 69.62) - open/close 3616 (time 1781283510.62 total 50.04 last 70.15) - open/close 4320 (time 1781283520.62 total 60.04 last 70.37) - open/close 5005 (time 1781283530.62 total 70.04 last 68.50) - open/close 5705 (time 1781283540.63 total 80.04 last 69.99) - open/close 6370 (time 1781283550.64 total 90.06 last 66.37) - open/close 7019 (time 1781283560.66 total 100.07 last 64.82) - open/close 7702 (time 1781283570.69 total 110.11 last 68.08) - open/close 8401 (time 1781283580.70 total 120.11 last 69.85) - open/close 9120 (time 1781283590.70 total 130.12 last 71.85) - open/close 9831 (time 1781283600.72 total 140.14 last 70.99) total: 10000 open/close in 142.79 seconds: 70.03 ops/second - unlinked 0 (time 1781283607 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341618 s, 3.1 MB/s fail_loc=0x80000204 Failing ost1 on oleg622-server Stopping /mnt/lustre-ost1 (opts:) on oleg622-server reboot facets: ost1 Failover ost1 to oleg622-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1781283766.19 total 34.02 last 0.00) total: 100 open/close in 36.20 seconds: 2.76 ops/second umount lustre on /mnt/lustre..... Stopping client oleg622-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg622-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg622-server stop mds service on oleg622-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg622-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing unload_modules_local oleg622-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg622-server: LNET ready to unload oleg622-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.