== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 14:58:55 (1776193135) Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) oleg425-server: oleg425-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg425-server: detected 4 online CPUs by sysfs oleg425-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg425-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock oleg425-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 478 (time 1776193273.38 total 10.00 last 47.78) - open/close 910 (time 1776193283.38 total 20.01 last 43.17) - open/close 1346 (time 1776193293.42 total 30.04 last 43.46) - open/close 1856 (time 1776193303.43 total 40.06 last 50.93) - open/close 2405 (time 1776193313.46 total 50.09 last 54.71) - open/close 3033 (time 1776193323.47 total 60.10 last 62.75) - open/close 3603 (time 1776193333.48 total 70.11 last 56.95) - open/close 4193 (time 1776193343.48 total 80.11 last 58.97) - open/close 4823 (time 1776193353.52 total 90.14 last 62.80) - open/close 5427 (time 1776193363.55 total 100.18 last 60.17) - open/close 6002 (time 1776193373.57 total 110.20 last 57.39) - open/close 6596 (time 1776193383.58 total 120.21 last 59.33) - open/close 7149 (time 1776193393.60 total 130.22 last 55.24) - open/close 7771 (time 1776193403.60 total 140.23 last 62.16) - open/close 8370 (time 1776193413.63 total 150.26 last 59.71) - open/close 9000 (time 1776193423.64 total 160.26 last 62.99) - open/close 9604 (time 1776193433.64 total 170.27 last 60.35) total: 10000 open/close in 177.31 seconds: 56.40 ops/second - unlinked 0 (time 1776193445 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 490 (time 1776193566.24 total 10.01 last 48.96) - open/close 900 (time 1776193576.25 total 20.02 last 40.95) - open/close 1313 (time 1776193586.27 total 30.04 last 41.24) - open/close 1861 (time 1776193596.28 total 40.04 last 54.75) - open/close 2426 (time 1776193606.29 total 50.06 last 56.42) - open/close 3003 (time 1776193616.30 total 60.06 last 57.67) - open/close 3622 (time 1776193626.30 total 70.07 last 61.87) - open/close 4238 (time 1776193636.30 total 80.07 last 61.59) - open/close 4805 (time 1776193646.31 total 90.07 last 56.69) - open/close 5367 (time 1776193656.32 total 100.08 last 56.14) - open/close 5923 (time 1776193666.33 total 110.09 last 55.55) - open/close 6469 (time 1776193676.35 total 120.11 last 54.49) - open/close 7023 (time 1776193686.36 total 130.13 last 55.32) - open/close 7638 (time 1776193696.37 total 140.14 last 61.41) - open/close 8184 (time 1776193706.39 total 150.16 last 54.51) - open/close 8737 (time 1776193716.40 total 160.17 last 55.22) - open/close 9281 (time 1776193726.41 total 170.18 last 54.34) - open/close 9889 (time 1776193736.43 total 180.20 last 60.69) total: 10000 open/close in 182.13 seconds: 54.91 ops/second - unlinked 0 (time 1776193743 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 642 (time 1776193855.82 total 10.00 last 64.17) - open/close 1194 (time 1776193865.84 total 20.02 last 55.10) - open/close 1826 (time 1776193875.85 total 30.04 last 63.11) - open/close 2445 (time 1776193885.86 total 40.05 last 61.83) - open/close 3096 (time 1776193895.87 total 50.05 last 65.05) - open/close 3767 (time 1776193905.90 total 60.08 last 66.92) - open/close 4313 (time 1776193915.95 total 70.13 last 54.32) - open/close 4896 (time 1776193925.96 total 80.15 last 58.22) - open/close 5554 (time 1776193935.98 total 90.16 last 65.73) - open/close 6119 (time 1776193945.98 total 100.16 last 56.48) - open/close 6684 (time 1776193955.99 total 110.17 last 56.46) - open/close 7207 (time 1776193965.99 total 120.17 last 52.28) - open/close 7807 (time 1776193976.00 total 130.19 last 59.92) - open/close 8351 (time 1776193986.00 total 140.19 last 54.39) - open/close 8935 (time 1776193996.02 total 150.20 last 58.34) - open/close 9556 (time 1776194006.02 total 160.20 last 62.06) total: 10000 open/close in 167.21 seconds: 59.81 ops/second - unlinked 0 (time 1776194018 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 565 (time 1776194132.07 total 10.00 last 56.47) - open/close 1084 (time 1776194142.08 total 20.02 last 51.83) - open/close 1577 (time 1776194152.10 total 30.03 last 49.22) - open/close 2123 (time 1776194162.11 total 40.04 last 54.56) - open/close 2732 (time 1776194172.12 total 50.05 last 60.85) - open/close 3368 (time 1776194182.12 total 60.06 last 63.56) - open/close 3957 (time 1776194192.12 total 70.06 last 58.90) - open/close 4591 (time 1776194202.12 total 80.06 last 63.38) - open/close 5220 (time 1776194212.13 total 90.06 last 62.89) - open/close 5832 (time 1776194222.13 total 100.07 last 61.17) - open/close 6407 (time 1776194232.14 total 110.07 last 57.47) - open/close 7012 (time 1776194242.14 total 120.07 last 60.49) - open/close 7636 (time 1776194252.14 total 130.08 last 62.38) - open/close 8243 (time 1776194262.16 total 140.09 last 60.59) - open/close 8852 (time 1776194272.17 total 150.10 last 60.84) - open/close 9421 (time 1776194282.18 total 160.12 last 56.84) total: 10000 open/close in 170.11 seconds: 58.79 ops/second - unlinked 0 (time 1776194297 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 599 (time 1776194406.70 total 10.01 last 59.82) - open/close 1233 (time 1776194416.72 total 20.03 last 63.28) - open/close 1842 (time 1776194426.72 total 30.04 last 60.87) - open/close 2479 (time 1776194436.73 total 40.04 last 63.67) - open/close 3117 (time 1776194446.74 total 50.05 last 63.75) - open/close 3817 (time 1776194456.74 total 60.05 last 69.98) - open/close 4476 (time 1776194466.74 total 70.06 last 65.86) - open/close 5066 (time 1776194476.76 total 80.08 last 58.89) - open/close 5654 (time 1776194486.76 total 90.08 last 58.80) - open/close 6230 (time 1776194496.77 total 100.09 last 57.56) - open/close 6851 (time 1776194506.78 total 110.09 last 62.06) - open/close 7415 (time 1776194516.78 total 120.10 last 56.37) - open/close 7970 (time 1776194526.79 total 130.11 last 55.47) - open/close 8526 (time 1776194536.80 total 140.12 last 55.54) - open/close 9105 (time 1776194546.81 total 150.13 last 57.82) - open/close 9749 (time 1776194556.83 total 160.14 last 64.31) total: 10000 open/close in 164.12 seconds: 60.93 ops/second - unlinked 0 (time 1776194564 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 660 (time 1776194673.63 total 10.01 last 65.91) - open/close 1292 (time 1776194683.64 total 20.02 last 63.14) - open/close 1910 (time 1776194693.65 total 30.03 last 61.74) - open/close 2558 (time 1776194703.67 total 40.06 last 64.65) - open/close 3149 (time 1776194713.68 total 50.07 last 59.04) - open/close 3812 (time 1776194723.68 total 60.07 last 66.30) - open/close 4478 (time 1776194733.69 total 70.07 last 66.57) - open/close 5135 (time 1776194743.71 total 80.09 last 65.58) - open/close 5797 (time 1776194753.72 total 90.10 last 66.12) - open/close 6492 (time 1776194763.74 total 100.12 last 69.38) - open/close 7168 (time 1776194773.74 total 110.12 last 67.56) - open/close 7756 (time 1776194783.74 total 120.12 last 58.80) - open/close 8362 (time 1776194793.76 total 130.14 last 60.51) - open/close 8976 (time 1776194803.77 total 140.15 last 61.31) - open/close 9594 (time 1776194813.78 total 150.16 last 61.75) total: 10000 open/close in 156.21 seconds: 64.01 ops/second - unlinked 0 (time 1776194823 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 654 (time 1776194929.18 total 10.01 last 65.36) - open/close 1273 (time 1776194939.19 total 20.01 last 61.88) - open/close 1895 (time 1776194949.19 total 30.01 last 62.19) - open/close 2499 (time 1776194959.19 total 40.01 last 60.37) - open/close 3103 (time 1776194969.20 total 50.02 last 60.37) - open/close 3729 (time 1776194979.21 total 60.03 last 62.54) - open/close 4375 (time 1776194989.21 total 70.03 last 64.60) - open/close 5020 (time 1776194999.21 total 80.03 last 64.46) - open/close 5593 (time 1776195009.21 total 90.03 last 57.30) - open/close 6219 (time 1776195019.22 total 100.04 last 62.54) - open/close 6785 (time 1776195029.24 total 110.06 last 56.53) - open/close 7318 (time 1776195039.26 total 120.08 last 53.17) - open/close 7943 (time 1776195049.26 total 130.08 last 62.50) - open/close 8593 (time 1776195059.27 total 140.09 last 64.94) - open/close 9204 (time 1776195069.28 total 150.10 last 61.07) - open/close 9866 (time 1776195079.29 total 160.11 last 66.09) total: 10000 open/close in 162.12 seconds: 61.68 ops/second - unlinked 0 (time 1776195085 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 603 (time 1776195195.44 total 10.01 last 60.27) - open/close 1254 (time 1776195205.46 total 20.02 last 64.98) - open/close 1920 (time 1776195215.46 total 30.03 last 66.58) - open/close 2622 (time 1776195225.46 total 40.03 last 70.19) - open/close 3272 (time 1776195235.47 total 50.04 last 64.95) - open/close 4030 (time 1776195245.48 total 60.05 last 75.72) - open/close 4767 (time 1776195255.49 total 70.06 last 73.60) - open/close 5527 (time 1776195265.50 total 80.07 last 75.95) - open/close 6223 (time 1776195275.51 total 90.08 last 69.48) - open/close 6847 (time 1776195285.52 total 100.09 last 62.38) - open/close 7479 (time 1776195295.52 total 110.09 last 63.15) - open/close 8058 (time 1776195305.52 total 120.09 last 57.90) - open/close 8666 (time 1776195315.53 total 130.10 last 60.74) - open/close 9332 (time 1776195325.54 total 140.11 last 66.58) - open/close 9988 (time 1776195335.54 total 150.11 last 65.58) total: 10000 open/close in 150.26 seconds: 66.55 ops/second - unlinked 0 (time 1776195339 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 602 (time 1776195446.44 total 10.00 last 60.18) - open/close 1229 (time 1776195456.45 total 20.01 last 62.65) - open/close 1856 (time 1776195466.45 total 30.01 last 62.69) - open/close 2539 (time 1776195476.46 total 40.02 last 68.23) - open/close 3142 (time 1776195486.47 total 50.03 last 60.27) - open/close 3800 (time 1776195496.47 total 60.03 last 65.78) - open/close 4446 (time 1776195506.49 total 70.04 last 64.52) - open/close 5086 (time 1776195516.49 total 80.05 last 63.99) - open/close 5672 (time 1776195526.49 total 90.05 last 58.58) - open/close 6337 (time 1776195536.50 total 100.06 last 66.44) - open/close 6938 (time 1776195546.50 total 110.06 last 60.07) - open/close 7524 (time 1776195556.52 total 120.08 last 58.53) - open/close 8177 (time 1776195566.52 total 130.08 last 65.26) - open/close 8762 (time 1776195576.55 total 140.10 last 58.37) - open/close 9408 (time 1776195586.55 total 150.11 last 64.55) total: 10000 open/close in 159.11 seconds: 62.85 ops/second - unlinked 0 (time 1776195600 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 626 (time 1776195703.08 total 10.01 last 62.54) - open/close 1244 (time 1776195713.09 total 20.02 last 61.75) - open/close 1872 (time 1776195723.09 total 30.02 last 62.77) - open/close 2495 (time 1776195733.10 total 40.02 last 62.29) - open/close 3165 (time 1776195743.10 total 50.03 last 66.98) - open/close 3858 (time 1776195753.11 total 60.04 last 69.19) - open/close 4508 (time 1776195763.12 total 70.05 last 64.99) - open/close 5150 (time 1776195773.14 total 80.07 last 64.05) - open/close 5787 (time 1776195783.14 total 90.07 last 63.67) - open/close 6386 (time 1776195793.15 total 100.08 last 59.88) - open/close 7016 (time 1776195803.15 total 110.08 last 62.98) - open/close 7606 (time 1776195813.16 total 120.09 last 58.96) - open/close 8224 (time 1776195823.17 total 130.09 last 61.75) - open/close 8808 (time 1776195833.17 total 140.10 last 58.37) - open/close 9226 (time 1776195843.18 total 150.11 last 41.77) - open/close 9640 (time 1776195853.19 total 160.12 last 41.34) total: 10000 open/close in 166.86 seconds: 59.93 ops/second - unlinked 0 (time 1776195864 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 569 (time 1776195972.48 total 10.01 last 56.85) - open/close 1158 (time 1776195982.49 total 20.01 last 58.88) - open/close 1878 (time 1776195992.50 total 30.02 last 71.94) - open/close 2578 (time 1776196002.50 total 40.03 last 69.96) - open/close 3297 (time 1776196012.51 total 50.03 last 71.89) - open/close 3971 (time 1776196022.51 total 60.03 last 67.38) - open/close 4693 (time 1776196032.52 total 70.04 last 72.12) - open/close 5363 (time 1776196042.54 total 80.06 last 66.89) - open/close 6037 (time 1776196052.54 total 90.06 last 67.37) - open/close 6734 (time 1776196062.54 total 100.07 last 69.67) - open/close 7464 (time 1776196072.55 total 110.08 last 72.93) - open/close 8151 (time 1776196082.56 total 120.08 last 68.65) - open/close 8841 (time 1776196092.57 total 130.10 last 68.92) - open/close 9486 (time 1776196102.58 total 140.10 last 64.48) total: 10000 open/close in 147.48 seconds: 67.81 ops/second - unlinked 0 (time 1776196113 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53946 s, 1.9 MB/s fail_loc=0x80000204 Failing ost1 on oleg425-server Stopping /mnt/lustre-ost1 (opts:) on oleg425-server reboot facets: ost1 Failover ost1 to oleg425-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776196286.27 total 33.92 last 0.00) total: 100 open/close in 36.33 seconds: 2.75 ops/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg425-server: LNET ready to unload oleg425-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.