== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 13:30:01 (1776187801) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) oleg319-server: oleg319-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 581 (time 1776187924.82 total 10.01 last 58.03) - open/close 1167 (time 1776187934.84 total 20.03 last 58.50) - open/close 1968 (time 1776187944.85 total 30.04 last 80.04) - open/close 2741 (time 1776187954.86 total 40.05 last 77.24) - open/close 3478 (time 1776187964.87 total 50.06 last 73.62) - open/close 4198 (time 1776187974.87 total 60.06 last 72.00) - open/close 4920 (time 1776187984.87 total 70.06 last 72.15) - open/close 5672 (time 1776187994.88 total 80.06 last 75.18) - open/close 6446 (time 1776188004.88 total 90.07 last 77.36) - open/close 7216 (time 1776188014.89 total 100.08 last 76.93) - open/close 7913 (time 1776188024.89 total 110.08 last 69.69) - open/close 8596 (time 1776188034.89 total 120.08 last 68.29) - open/close 9285 (time 1776188044.90 total 130.09 last 68.87) - open/close 9924 (time 1776188054.91 total 140.10 last 63.83) total: 10000 open/close in 141.19 seconds: 70.83 ops/second - unlinked 0 (time 1776188060 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 720 (time 1776188170.04 total 10.00 last 71.98) - open/close 1419 (time 1776188180.06 total 20.02 last 69.75) - open/close 2164 (time 1776188190.06 total 30.03 last 74.48) - open/close 2984 (time 1776188200.07 total 40.04 last 81.91) - open/close 3707 (time 1776188210.07 total 50.04 last 72.28) - open/close 4434 (time 1776188220.08 total 60.05 last 72.64) - open/close 5142 (time 1776188230.08 total 70.05 last 70.79) - open/close 5818 (time 1776188240.10 total 80.06 last 67.52) - open/close 6502 (time 1776188250.11 total 90.07 last 68.33) - open/close 7158 (time 1776188260.12 total 100.09 last 65.50) - open/close 7815 (time 1776188270.12 total 110.09 last 65.68) - open/close 8518 (time 1776188280.13 total 120.10 last 70.24) - open/close 9207 (time 1776188290.14 total 130.10 last 68.87) - open/close 9936 (time 1776188300.15 total 140.11 last 72.82) total: 10000 open/close in 140.92 seconds: 70.96 ops/second - unlinked 0 (time 1776188304 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 692 (time 1776188408.03 total 10.02 last 69.08) - open/close 1404 (time 1776188418.04 total 20.03 last 71.14) - open/close 2166 (time 1776188428.04 total 30.03 last 76.15) - open/close 2912 (time 1776188438.05 total 40.04 last 74.54) - open/close 3638 (time 1776188448.06 total 50.05 last 72.54) - open/close 4327 (time 1776188458.07 total 60.05 last 68.86) - open/close 5040 (time 1776188468.07 total 70.06 last 71.27) - open/close 5683 (time 1776188478.07 total 80.06 last 64.29) - open/close 6327 (time 1776188488.08 total 90.07 last 64.33) - open/close 6994 (time 1776188498.09 total 100.08 last 66.64) - open/close 7624 (time 1776188508.10 total 110.09 last 62.92) - open/close 8299 (time 1776188518.11 total 120.10 last 67.47) - open/close 8976 (time 1776188528.11 total 130.10 last 67.68) - open/close 9612 (time 1776188538.11 total 140.10 last 63.59) total: 10000 open/close in 145.24 seconds: 68.85 ops/second - unlinked 0 (time 1776188547 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 762 (time 1776188646.89 total 10.01 last 76.15) - open/close 1423 (time 1776188656.89 total 20.01 last 66.09) - open/close 2109 (time 1776188666.90 total 30.02 last 68.53) - open/close 2817 (time 1776188676.91 total 40.03 last 70.71) - open/close 3602 (time 1776188686.91 total 50.03 last 78.50) - open/close 4324 (time 1776188696.92 total 60.04 last 72.11) - open/close 5064 (time 1776188706.93 total 70.05 last 73.95) - open/close 5813 (time 1776188716.94 total 80.06 last 74.84) - open/close 6562 (time 1776188726.94 total 90.06 last 74.87) - open/close 7280 (time 1776188736.96 total 100.08 last 71.70) - open/close 8011 (time 1776188746.96 total 110.08 last 73.07) - open/close 8736 (time 1776188756.97 total 120.09 last 72.41) - open/close 9476 (time 1776188766.98 total 130.10 last 73.96) total: 10000 open/close in 137.73 seconds: 72.61 ops/second - unlinked 0 (time 1776188778 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 787 (time 1776188872.06 total 10.00 last 78.67) - open/close 1487 (time 1776188882.07 total 20.01 last 69.96) - open/close 2127 (time 1776188892.09 total 30.03 last 63.90) - open/close 2835 (time 1776188902.10 total 40.04 last 70.73) - open/close 3510 (time 1776188912.11 total 50.04 last 67.44) - open/close 4222 (time 1776188922.11 total 60.05 last 71.19) - open/close 4905 (time 1776188932.11 total 70.05 last 68.25) - open/close 5622 (time 1776188942.12 total 80.06 last 71.64) - open/close 6302 (time 1776188952.14 total 90.08 last 67.91) - open/close 6991 (time 1776188962.14 total 100.08 last 68.87) - open/close 7749 (time 1776188972.16 total 110.10 last 75.65) - open/close 8372 (time 1776188982.17 total 120.11 last 62.23) - open/close 9064 (time 1776188992.17 total 130.11 last 69.19) - open/close 9747 (time 1776189002.17 total 140.11 last 68.29) total: 10000 open/close in 143.80 seconds: 69.54 ops/second - unlinked 0 (time 1776189009 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 673 (time 1776189114.53 total 10.00 last 67.29) - open/close 1309 (time 1776189124.54 total 20.01 last 63.55) - open/close 1931 (time 1776189134.54 total 30.01 last 62.18) - open/close 2614 (time 1776189144.55 total 40.02 last 68.27) - open/close 3201 (time 1776189154.55 total 50.02 last 58.67) - open/close 3804 (time 1776189164.56 total 60.03 last 60.28) - open/close 4449 (time 1776189174.56 total 70.03 last 64.48) - open/close 5123 (time 1776189184.57 total 80.04 last 67.34) - open/close 5822 (time 1776189194.57 total 90.04 last 69.87) - open/close 6498 (time 1776189204.61 total 100.07 last 67.38) - open/close 7178 (time 1776189214.62 total 110.09 last 67.92) - open/close 7905 (time 1776189224.63 total 120.10 last 72.59) - open/close 8566 (time 1776189234.64 total 130.11 last 66.05) - open/close 9274 (time 1776189244.65 total 140.12 last 70.70) - open/close 9959 (time 1776189254.66 total 150.13 last 68.46) total: 10000 open/close in 150.66 seconds: 66.37 ops/second - unlinked 0 (time 1776189259 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 784 (time 1776189350.73 total 10.01 last 78.33) - open/close 1489 (time 1776189360.73 total 20.01 last 70.49) - open/close 2187 (time 1776189370.74 total 30.01 last 69.78) - open/close 2886 (time 1776189380.75 total 40.03 last 69.81) - open/close 3575 (time 1776189390.76 total 50.04 last 68.82) - open/close 4286 (time 1776189400.77 total 60.04 last 71.05) - open/close 4931 (time 1776189410.77 total 70.05 last 64.46) - open/close 5603 (time 1776189420.78 total 80.06 last 67.15) - open/close 6357 (time 1776189430.78 total 90.06 last 75.38) - open/close 7005 (time 1776189440.81 total 100.09 last 64.63) - open/close 7688 (time 1776189450.81 total 110.09 last 68.26) - open/close 8401 (time 1776189460.82 total 120.09 last 71.29) - open/close 9026 (time 1776189470.83 total 130.11 last 62.39) - open/close 9677 (time 1776189480.84 total 140.12 last 65.05) total: 10000 open/close in 145.39 seconds: 68.78 ops/second - unlinked 0 (time 1776189490 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 658 (time 1776189591.44 total 10.01 last 65.71) - open/close 1190 (time 1776189601.44 total 20.02 last 53.16) - open/close 1818 (time 1776189611.46 total 30.03 last 62.73) - open/close 2488 (time 1776189621.46 total 40.04 last 66.97) - open/close 3118 (time 1776189631.46 total 50.04 last 63.00) - open/close 3750 (time 1776189641.47 total 60.05 last 63.12) - open/close 4446 (time 1776189651.49 total 70.07 last 69.47) - open/close 5065 (time 1776189661.50 total 80.08 last 61.84) - open/close 5669 (time 1776189671.51 total 90.09 last 60.33) - open/close 6316 (time 1776189681.53 total 100.11 last 64.60) - open/close 6986 (time 1776189691.53 total 110.11 last 66.98) - open/close 7654 (time 1776189701.54 total 120.12 last 66.75) - open/close 8312 (time 1776189711.54 total 130.12 last 65.80) - open/close 8947 (time 1776189721.55 total 140.12 last 63.46) - open/close 9533 (time 1776189731.55 total 150.13 last 58.55) total: 10000 open/close in 159.41 seconds: 62.73 ops/second - unlinked 0 (time 1776189746 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 610 (time 1776189853.70 total 10.01 last 60.96) - open/close 1294 (time 1776189863.71 total 20.02 last 68.32) - open/close 1940 (time 1776189873.72 total 30.03 last 64.52) - open/close 2636 (time 1776189883.73 total 40.03 last 69.58) - open/close 3287 (time 1776189893.73 total 50.04 last 65.04) - open/close 3983 (time 1776189903.74 total 60.05 last 69.58) - open/close 4597 (time 1776189913.76 total 70.07 last 61.26) - open/close 5237 (time 1776189923.77 total 80.07 last 63.97) - open/close 5861 (time 1776189933.78 total 90.09 last 62.31) - open/close 6438 (time 1776189943.78 total 100.09 last 57.69) - open/close 7081 (time 1776189953.79 total 110.09 last 64.27) - open/close 7739 (time 1776189963.80 total 120.10 last 65.73) - open/close 8388 (time 1776189973.82 total 130.13 last 64.74) - open/close 9058 (time 1776189983.84 total 140.14 last 66.91) - open/close 9696 (time 1776189993.84 total 150.15 last 63.75) total: 10000 open/close in 154.89 seconds: 64.56 ops/second - unlinked 0 (time 1776190002 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 686 (time 1776190098.95 total 10.01 last 68.50) - open/close 1376 (time 1776190108.96 total 20.02 last 68.93) - open/close 2070 (time 1776190118.96 total 30.02 last 69.40) - open/close 2781 (time 1776190128.98 total 40.04 last 70.98) - open/close 3491 (time 1776190138.98 total 50.05 last 70.97) - open/close 4199 (time 1776190148.99 total 60.06 last 70.73) - open/close 4928 (time 1776190159.00 total 70.06 last 72.85) - open/close 5666 (time 1776190169.01 total 80.07 last 73.74) - open/close 6374 (time 1776190179.03 total 90.09 last 70.65) - open/close 7122 (time 1776190189.03 total 100.10 last 74.77) - open/close 7885 (time 1776190199.03 total 110.10 last 76.29) - open/close 8642 (time 1776190209.03 total 120.10 last 75.70) - open/close 9433 (time 1776190219.04 total 130.10 last 79.07) total: 10000 open/close in 137.08 seconds: 72.95 ops/second - unlinked 0 (time 1776190228 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 618 (time 1776190321.75 total 10.00 last 61.78) - open/close 1286 (time 1776190331.77 total 20.01 last 66.73) - open/close 2054 (time 1776190341.77 total 30.02 last 76.73) - open/close 2744 (time 1776190351.78 total 40.03 last 68.93) - open/close 3468 (time 1776190361.79 total 50.04 last 72.36) - open/close 4259 (time 1776190371.80 total 60.05 last 79.04) - open/close 5067 (time 1776190381.81 total 70.06 last 80.73) - open/close 5918 (time 1776190391.81 total 80.06 last 85.06) - open/close 6737 (time 1776190401.82 total 90.07 last 81.85) - open/close 7471 (time 1776190411.84 total 100.08 last 73.27) - open/close 8145 (time 1776190421.84 total 110.09 last 67.35) - open/close 8714 (time 1776190431.85 total 120.10 last 56.84) - open/close 9244 (time 1776190441.86 total 130.11 last 52.97) - open/close 9896 (time 1776190451.87 total 140.12 last 65.16) total: 10000 open/close in 141.67 seconds: 70.58 ops/second - unlinked 0 (time 1776190457 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246668 s, 4.3 MB/s fail_loc=0x80000204 Failing ost1 on oleg319-server Stopping /mnt/lustre-ost1 (opts:) on oleg319-server reboot facets: ost1 Failover ost1 to oleg319-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776190628.08 total 34.02 last 0.00) total: 100 open/close in 36.30 seconds: 2.75 ops/second umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local oleg319-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg319-server: LNET ready to unload oleg319-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.