== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 16:49:53 (1776199793) Stopping clients: oleg454-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg454-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 oleg454-server: oleg454-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: detected 4 online CPUs by sysfs oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg454-server: detected 4 online CPUs by sysfs oleg454-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg454-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg454-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg454-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock oleg454-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 583 (time 1776199976.19 total 10.01 last 58.26) - open/close 1140 (time 1776199986.20 total 20.01 last 55.69) - open/close 1784 (time 1776199996.20 total 30.01 last 64.39) - open/close 2389 (time 1776200006.20 total 40.02 last 60.45) - open/close 3004 (time 1776200016.22 total 50.03 last 61.41) - open/close 3520 (time 1776200026.22 total 60.04 last 51.58) - open/close 4101 (time 1776200036.23 total 70.04 last 58.05) - open/close 4763 (time 1776200046.23 total 80.05 last 66.18) - open/close 5358 (time 1776200056.25 total 90.06 last 59.40) - open/close 5964 (time 1776200066.26 total 100.07 last 60.57) - open/close 6558 (time 1776200076.26 total 110.07 last 59.40) - open/close 7217 (time 1776200086.26 total 120.07 last 65.88) - open/close 7863 (time 1776200096.27 total 130.09 last 64.50) - open/close 8481 (time 1776200106.27 total 140.09 last 61.79) - open/close 9141 (time 1776200116.28 total 150.09 last 65.98) - open/close 9820 (time 1776200126.29 total 160.10 last 67.81) total: 10000 open/close in 163.43 seconds: 61.19 ops/second - unlinked 0 (time 1776200135 ; total 1 ; last 1) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 617 (time 1776200248.93 total 10.01 last 61.65) - open/close 1239 (time 1776200258.94 total 20.01 last 62.17) - open/close 1871 (time 1776200268.94 total 30.01 last 63.18) - open/close 2542 (time 1776200278.94 total 40.02 last 67.09) - open/close 3177 (time 1776200288.95 total 50.03 last 63.43) - open/close 3783 (time 1776200298.96 total 60.03 last 60.57) - open/close 4360 (time 1776200308.98 total 70.05 last 57.59) - open/close 4924 (time 1776200319.00 total 80.07 last 56.29) - open/close 5604 (time 1776200329.00 total 90.07 last 67.97) - open/close 6267 (time 1776200339.00 total 100.08 last 66.28) - open/close 6909 (time 1776200349.02 total 110.09 last 64.13) - open/close 7573 (time 1776200359.02 total 120.10 last 66.35) - open/close 8243 (time 1776200369.03 total 130.11 last 66.92) - open/close 8882 (time 1776200379.04 total 140.11 last 63.85) - open/close 9457 (time 1776200389.05 total 150.12 last 57.46) - open/close 9982 (time 1776200399.05 total 160.13 last 52.48) total: 10000 open/close in 160.49 seconds: 62.31 ops/second - unlinked 0 (time 1776200405 ; total 0 ; last 0) total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second - open/close 663 (time 1776200526.63 total 10.01 last 66.22) - open/close 1280 (time 1776200536.63 total 20.02 last 61.66) - open/close 1914 (time 1776200546.64 total 30.03 last 63.34) - open/close 2617 (time 1776200556.65 total 40.03 last 70.26) - open/close 3178 (time 1776200566.65 total 50.03 last 56.10) - open/close 3697 (time 1776200576.65 total 60.04 last 51.87) - open/close 4309 (time 1776200586.67 total 70.05 last 61.12) - open/close 4994 (time 1776200596.67 total 80.05 last 68.50) - open/close 5640 (time 1776200606.69 total 90.08 last 64.43) - open/close 6284 (time 1776200616.70 total 100.08 last 64.39) - open/close 6854 (time 1776200626.71 total 110.10 last 56.90) - open/close 7467 (time 1776200636.74 total 120.12 last 61.16) - open/close 8115 (time 1776200646.75 total 130.13 last 64.75) - open/close 8759 (time 1776200656.75 total 140.14 last 64.35) - open/close 9409 (time 1776200666.76 total 150.14 last 64.98) - open/close 9946 (time 1776200676.76 total 160.15 last 53.66) total: 10000 open/close in 161.19 seconds: 62.04 ops/second - unlinked 0 (time 1776200683 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 557 (time 1776200797.24 total 10.02 last 55.59) - open/close 1003 (time 1776200807.24 total 20.02 last 44.59) - open/close 1482 (time 1776200817.26 total 30.05 last 47.79) - open/close 2201 (time 1776200827.27 total 40.05 last 71.86) - open/close 2841 (time 1776200837.27 total 50.06 last 63.95) - open/close 3427 (time 1776200847.28 total 60.07 last 58.54) - open/close 4029 (time 1776200857.29 total 70.07 last 60.19) - open/close 4690 (time 1776200867.29 total 80.07 last 66.09) - open/close 5307 (time 1776200877.31 total 90.09 last 61.58) - open/close 5976 (time 1776200887.32 total 100.10 last 66.83) - open/close 6600 (time 1776200897.32 total 110.11 last 62.37) - open/close 7248 (time 1776200907.32 total 120.11 last 64.80) - open/close 7885 (time 1776200917.33 total 130.11 last 63.67) - open/close 8504 (time 1776200927.34 total 140.12 last 61.85) - open/close 9130 (time 1776200937.34 total 150.13 last 62.57) - open/close 9723 (time 1776200947.35 total 160.14 last 59.25) total: 10000 open/close in 164.18 seconds: 60.91 ops/second - unlinked 0 (time 1776200955 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 622 (time 1776201062.87 total 10.00 last 62.18) - open/close 1295 (time 1776201072.89 total 20.02 last 67.21) - open/close 1948 (time 1776201082.91 total 30.03 last 65.19) - open/close 2604 (time 1776201092.92 total 40.04 last 65.53) - open/close 3252 (time 1776201102.92 total 50.05 last 64.74) - open/close 3904 (time 1776201112.94 total 60.07 last 65.10) - open/close 4519 (time 1776201122.94 total 70.07 last 61.47) - open/close 5152 (time 1776201132.95 total 80.08 last 63.26) - open/close 5729 (time 1776201142.95 total 90.08 last 57.69) - open/close 6350 (time 1776201152.97 total 100.10 last 62.01) - open/close 6982 (time 1776201162.98 total 110.11 last 63.11) - open/close 7552 (time 1776201172.99 total 120.12 last 56.97) - open/close 8198 (time 1776201182.99 total 130.12 last 64.56) - open/close 8865 (time 1776201193.02 total 140.14 last 66.55) - open/close 9534 (time 1776201203.02 total 150.15 last 66.89) total: 10000 open/close in 157.04 seconds: 63.68 ops/second - unlinked 0 (time 1776201214 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 626 (time 1776201323.17 total 10.01 last 62.52) - open/close 1271 (time 1776201333.18 total 20.02 last 64.44) - open/close 1899 (time 1776201343.18 total 30.02 last 62.80) - open/close 2551 (time 1776201353.18 total 40.03 last 65.19) - open/close 3186 (time 1776201363.19 total 50.04 last 63.42) - open/close 3887 (time 1776201373.20 total 60.05 last 70.05) - open/close 4536 (time 1776201383.22 total 70.07 last 64.78) - open/close 5178 (time 1776201393.24 total 80.08 last 64.10) - open/close 5846 (time 1776201403.24 total 90.08 last 66.78) - open/close 6488 (time 1776201413.25 total 100.09 last 64.15) - open/close 7181 (time 1776201423.25 total 110.10 last 69.25) - open/close 7811 (time 1776201433.26 total 120.10 last 62.99) - open/close 8399 (time 1776201443.26 total 130.10 last 58.78) - open/close 9051 (time 1776201453.26 total 140.11 last 65.18) - open/close 9714 (time 1776201463.28 total 150.13 last 66.16) total: 10000 open/close in 154.29 seconds: 64.82 ops/second - unlinked 0 (time 1776201472 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 683 (time 1776201577.25 total 10.00 last 68.30) - open/close 1328 (time 1776201587.26 total 20.01 last 64.40) - open/close 1998 (time 1776201597.26 total 30.02 last 66.99) - open/close 2668 (time 1776201607.28 total 40.04 last 66.86) - open/close 3311 (time 1776201617.29 total 50.04 last 64.27) - open/close 3974 (time 1776201627.30 total 60.06 last 66.21) - open/close 4623 (time 1776201637.31 total 70.06 last 64.87) - open/close 5197 (time 1776201647.31 total 80.06 last 57.39) - open/close 5801 (time 1776201657.32 total 90.08 last 60.31) - open/close 6451 (time 1776201667.33 total 100.09 last 64.95) - open/close 7039 (time 1776201677.33 total 110.09 last 58.79) - open/close 7634 (time 1776201687.34 total 120.10 last 59.46) - open/close 8295 (time 1776201697.35 total 130.10 last 66.07) - open/close 8920 (time 1776201707.35 total 140.11 last 62.48) - open/close 9528 (time 1776201717.37 total 150.13 last 60.67) total: 10000 open/close in 158.20 seconds: 63.21 ops/second - unlinked 0 (time 1776201730 ; total 1 ; last 1) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 594 (time 1776201845.90 total 10.02 last 59.30) - open/close 1228 (time 1776201855.90 total 20.02 last 63.38) - open/close 1872 (time 1776201865.92 total 30.04 last 64.30) - open/close 2461 (time 1776201875.92 total 40.04 last 58.89) - open/close 3057 (time 1776201885.97 total 50.09 last 59.29) - open/close 3688 (time 1776201895.98 total 60.09 last 63.09) - open/close 4270 (time 1776201905.98 total 70.09 last 58.19) - open/close 4854 (time 1776201915.98 total 80.10 last 58.38) - open/close 5518 (time 1776201925.99 total 90.11 last 66.34) - open/close 6123 (time 1776201936.00 total 100.11 last 60.45) - open/close 6770 (time 1776201946.01 total 110.12 last 64.64) - open/close 7426 (time 1776201956.05 total 120.17 last 65.33) - open/close 8009 (time 1776201966.08 total 130.19 last 58.15) - open/close 8608 (time 1776201976.09 total 140.20 last 59.84) - open/close 9285 (time 1776201986.09 total 150.21 last 67.66) - open/close 9976 (time 1776201996.09 total 160.21 last 69.10) total: 10000 open/close in 160.53 seconds: 62.30 ops/second - unlinked 0 (time 1776202000 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 669 (time 1776202105.74 total 10.01 last 66.84) - open/close 1284 (time 1776202115.75 total 20.02 last 61.45) - open/close 1835 (time 1776202125.75 total 30.03 last 55.06) - open/close 2454 (time 1776202135.78 total 40.05 last 61.77) - open/close 3074 (time 1776202145.78 total 50.05 last 61.98) - open/close 3714 (time 1776202155.78 total 60.05 last 63.98) - open/close 4283 (time 1776202165.78 total 70.06 last 56.88) - open/close 4841 (time 1776202175.79 total 80.07 last 55.75) - open/close 5444 (time 1776202185.80 total 90.07 last 60.24) - open/close 6145 (time 1776202195.82 total 100.09 last 70.00) - open/close 6752 (time 1776202205.83 total 110.10 last 60.66) - open/close 7406 (time 1776202215.83 total 120.10 last 65.36) - open/close 8051 (time 1776202225.83 total 130.10 last 64.50) - open/close 8732 (time 1776202235.84 total 140.11 last 68.05) - open/close 9371 (time 1776202245.84 total 150.11 last 63.89) total: 10000 open/close in 159.68 seconds: 62.63 ops/second - unlinked 0 (time 1776202259 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 627 (time 1776202362.16 total 10.00 last 62.68) - open/close 1313 (time 1776202372.17 total 20.01 last 68.53) - open/close 1968 (time 1776202382.17 total 30.01 last 65.50) - open/close 2629 (time 1776202392.17 total 40.02 last 66.07) - open/close 3277 (time 1776202402.18 total 50.03 last 64.75) - open/close 3908 (time 1776202412.19 total 60.03 last 63.05) - open/close 4567 (time 1776202422.20 total 70.04 last 65.84) - open/close 5191 (time 1776202432.23 total 80.07 last 62.21) - open/close 5672 (time 1776202442.23 total 90.08 last 48.08) - open/close 6243 (time 1776202452.24 total 100.08 last 57.07) - open/close 6853 (time 1776202462.24 total 110.09 last 60.98) - open/close 7533 (time 1776202472.27 total 120.11 last 67.81) - open/close 8171 (time 1776202482.27 total 130.12 last 63.79) - open/close 8794 (time 1776202492.28 total 140.13 last 62.24) - open/close 9438 (time 1776202502.28 total 150.13 last 64.38) total: 10000 open/close in 159.58 seconds: 62.66 ops/second - unlinked 0 (time 1776202516 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 592 (time 1776202624.45 total 10.01 last 59.11) - open/close 1218 (time 1776202634.46 total 20.03 last 62.53) - open/close 1862 (time 1776202644.46 total 30.03 last 64.37) - open/close 2530 (time 1776202654.47 total 40.04 last 66.74) - open/close 3188 (time 1776202664.48 total 50.05 last 65.73) - open/close 3891 (time 1776202674.49 total 60.06 last 70.24) - open/close 4567 (time 1776202684.51 total 70.07 last 67.51) - open/close 5263 (time 1776202694.51 total 80.08 last 69.56) - open/close 5994 (time 1776202704.51 total 90.08 last 73.08) - open/close 6682 (time 1776202714.53 total 100.09 last 68.72) - open/close 7326 (time 1776202724.54 total 110.10 last 64.33) - open/close 7974 (time 1776202734.55 total 120.12 last 64.72) - open/close 8656 (time 1776202744.56 total 130.13 last 68.14) - open/close 9288 (time 1776202754.57 total 140.13 last 63.15) - open/close 9958 (time 1776202764.57 total 150.14 last 66.96) total: 10000 open/close in 150.59 seconds: 66.41 ops/second - unlinked 0 (time 1776202769 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4168 82424 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394503 s, 2.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg454-server Stopping /mnt/lustre-ost1 (opts:) on oleg454-server reboot facets: ost1 Failover ost1 to oleg454-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1776202946.43 total 29.75 last 0.00) total: 100 open/close in 32.63 seconds: 3.06 ops/second umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg454-server: LNET ready to unload oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.