== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 14:06:01 (1776189961) Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: oleg435-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg435-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 754 (time 1776190135.58 total 10.00 last 75.40) - open/close 1527 (time 1776190145.59 total 20.01 last 77.22) - open/close 2216 (time 1776190155.59 total 30.02 last 68.87) - open/close 2941 (time 1776190165.60 total 40.03 last 72.43) - open/close 3641 (time 1776190175.61 total 50.03 last 69.95) - open/close 4449 (time 1776190185.62 total 60.05 last 80.67) - open/close 5192 (time 1776190195.63 total 70.05 last 74.26) - open/close 5938 (time 1776190205.64 total 80.07 last 74.48) - open/close 6675 (time 1776190215.64 total 90.07 last 73.70) - open/close 7336 (time 1776190225.65 total 100.07 last 66.08) - open/close 8022 (time 1776190235.66 total 110.09 last 68.51) - open/close 8647 (time 1776190245.67 total 120.10 last 62.42) - open/close 9326 (time 1776190255.69 total 130.12 last 67.79) - open/close 9973 (time 1776190265.70 total 140.13 last 64.64) total: 10000 open/close in 140.43 seconds: 71.21 ops/second - unlinked 0 (time 1776190270 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 771 (time 1776190365.05 total 10.00 last 77.07) - open/close 1551 (time 1776190375.06 total 20.01 last 77.92) - open/close 2347 (time 1776190385.08 total 30.04 last 79.40) - open/close 3122 (time 1776190395.09 total 40.05 last 77.40) - open/close 3887 (time 1776190405.11 total 50.06 last 76.40) - open/close 4512 (time 1776190415.12 total 60.07 last 62.44) - open/close 5231 (time 1776190425.12 total 70.08 last 71.88) - open/close 5981 (time 1776190435.13 total 80.09 last 74.93) - open/close 6679 (time 1776190445.13 total 90.09 last 69.77) - open/close 7497 (time 1776190455.13 total 100.09 last 81.79) - open/close 8219 (time 1776190465.13 total 110.09 last 72.19) - open/close 8913 (time 1776190475.15 total 120.11 last 69.30) - open/close 9649 (time 1776190485.16 total 130.11 last 73.55) total: 10000 open/close in 134.40 seconds: 74.40 ops/second - unlinked 0 (time 1776190493 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 650 (time 1776190593.63 total 10.01 last 64.94) - open/close 1352 (time 1776190603.63 total 20.01 last 70.19) - open/close 2107 (time 1776190613.64 total 30.01 last 75.48) - open/close 2877 (time 1776190623.65 total 40.03 last 76.88) - open/close 3648 (time 1776190633.66 total 50.04 last 77.03) - open/close 4355 (time 1776190643.66 total 60.04 last 70.67) - open/close 5133 (time 1776190653.67 total 70.05 last 77.74) - open/close 5875 (time 1776190663.67 total 80.05 last 74.19) - open/close 6627 (time 1776190673.68 total 90.05 last 75.17) - open/close 7273 (time 1776190683.69 total 100.07 last 64.52) - open/close 7982 (time 1776190693.70 total 110.07 last 70.84) - open/close 8731 (time 1776190703.70 total 120.08 last 74.89) - open/close 9497 (time 1776190713.70 total 130.08 last 76.58) total: 10000 open/close in 137.19 seconds: 72.89 ops/second - unlinked 0 (time 1776190725 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 758 (time 1776190825.43 total 10.02 last 75.64) - open/close 1511 (time 1776190835.44 total 20.03 last 75.25) - open/close 2116 (time 1776190845.46 total 30.05 last 60.34) - open/close 2733 (time 1776190855.47 total 40.06 last 61.68) - open/close 3360 (time 1776190865.47 total 50.06 last 62.68) - open/close 4137 (time 1776190875.47 total 60.06 last 77.69) - open/close 4922 (time 1776190885.48 total 70.07 last 78.41) - open/close 5715 (time 1776190895.49 total 80.08 last 79.26) - open/close 6505 (time 1776190905.50 total 90.09 last 78.92) - open/close 7270 (time 1776190915.50 total 100.09 last 76.49) - open/close 8023 (time 1776190925.50 total 110.09 last 75.28) - open/close 8818 (time 1776190935.51 total 120.10 last 79.43) - open/close 9522 (time 1776190945.51 total 130.10 last 70.39) total: 10000 open/close in 136.42 seconds: 73.30 ops/second - unlinked 0 (time 1776190956 ; total 1 ; last 1) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 765 (time 1776191054.27 total 10.00 last 76.47) - open/close 1494 (time 1776191064.29 total 20.03 last 72.73) - open/close 2293 (time 1776191074.30 total 30.03 last 79.87) - open/close 3051 (time 1776191084.31 total 40.04 last 75.73) - open/close 3795 (time 1776191094.31 total 50.04 last 74.40) - open/close 4531 (time 1776191104.31 total 60.04 last 73.57) - open/close 5230 (time 1776191114.32 total 70.05 last 69.84) - open/close 6059 (time 1776191124.34 total 80.07 last 82.74) - open/close 6814 (time 1776191134.37 total 90.10 last 75.29) - open/close 7530 (time 1776191144.37 total 100.10 last 71.58) - open/close 8231 (time 1776191154.38 total 110.11 last 70.02) - open/close 9007 (time 1776191164.38 total 120.12 last 77.58) - open/close 9778 (time 1776191174.39 total 130.12 last 77.08) total: 10000 open/close in 132.90 seconds: 75.24 ops/second - unlinked 0 (time 1776191181 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 755 (time 1776191279.32 total 10.01 last 75.44) - open/close 1561 (time 1776191289.32 total 20.01 last 80.57) - open/close 2290 (time 1776191299.33 total 30.02 last 72.84) - open/close 3036 (time 1776191309.34 total 40.03 last 74.55) - open/close 3829 (time 1776191319.34 total 50.03 last 79.30) - open/close 4559 (time 1776191329.36 total 60.04 last 72.87) - open/close 5279 (time 1776191339.37 total 70.06 last 71.90) - open/close 6028 (time 1776191349.38 total 80.06 last 74.86) - open/close 6792 (time 1776191359.38 total 90.07 last 76.38) - open/close 7625 (time 1776191369.39 total 100.07 last 83.23) - open/close 8385 (time 1776191379.40 total 110.09 last 75.88) - open/close 9128 (time 1776191389.42 total 120.10 last 74.21) - open/close 9836 (time 1776191399.42 total 130.11 last 70.77) total: 10000 open/close in 132.40 seconds: 75.53 ops/second - unlinked 0 (time 1776191407 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 702 (time 1776191508.11 total 10.01 last 70.11) - open/close 1484 (time 1776191518.12 total 20.02 last 78.14) - open/close 2314 (time 1776191528.13 total 30.03 last 82.93) - open/close 2988 (time 1776191538.15 total 40.05 last 67.26) - open/close 3667 (time 1776191548.16 total 50.06 last 67.83) - open/close 4230 (time 1776191558.16 total 60.06 last 56.28) - open/close 4901 (time 1776191568.17 total 70.07 last 67.04) - open/close 5622 (time 1776191578.17 total 80.07 last 72.09) - open/close 6366 (time 1776191588.17 total 90.08 last 74.37) - open/close 6947 (time 1776191598.18 total 100.08 last 58.09) - open/close 7602 (time 1776191608.18 total 110.08 last 65.49) - open/close 8281 (time 1776191618.18 total 120.08 last 67.89) - open/close 8988 (time 1776191628.19 total 130.09 last 70.64) - open/close 9686 (time 1776191638.21 total 140.11 last 69.69) total: 10000 open/close in 145.14 seconds: 68.90 ops/second - unlinked 0 (time 1776191647 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 628 (time 1776191743.12 total 10.02 last 62.66) - open/close 1346 (time 1776191753.13 total 20.03 last 71.75) - open/close 2080 (time 1776191763.13 total 30.03 last 73.40) - open/close 2867 (time 1776191773.14 total 40.03 last 78.67) - open/close 3645 (time 1776191783.15 total 50.04 last 77.71) - open/close 4459 (time 1776191793.17 total 60.07 last 81.21) - open/close 5231 (time 1776191803.20 total 70.10 last 76.95) - open/close 6036 (time 1776191813.21 total 80.11 last 80.44) - open/close 6742 (time 1776191823.22 total 90.12 last 70.51) - open/close 7426 (time 1776191833.25 total 100.14 last 68.24) - open/close 8163 (time 1776191843.25 total 110.15 last 73.69) - open/close 8813 (time 1776191853.25 total 120.15 last 64.98) - open/close 9439 (time 1776191863.27 total 130.16 last 62.50) total: 10000 open/close in 137.94 seconds: 72.50 ops/second - unlinked 0 (time 1776191875 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 763 (time 1776191970.01 total 10.01 last 76.25) - open/close 1479 (time 1776191980.03 total 20.02 last 71.48) - open/close 2104 (time 1776191990.04 total 30.04 last 62.40) - open/close 2826 (time 1776192000.05 total 40.04 last 72.17) - open/close 3374 (time 1776192010.05 total 50.05 last 54.78) - open/close 4116 (time 1776192020.05 total 60.05 last 74.19) - open/close 4867 (time 1776192030.06 total 70.05 last 75.06) - open/close 5663 (time 1776192040.07 total 80.07 last 79.50) - open/close 6423 (time 1776192050.08 total 90.07 last 75.95) - open/close 7087 (time 1776192060.09 total 100.08 last 66.34) - open/close 7835 (time 1776192070.09 total 110.08 last 74.78) - open/close 8638 (time 1776192080.09 total 120.09 last 80.28) - open/close 9394 (time 1776192090.09 total 130.09 last 75.59) total: 10000 open/close in 137.90 seconds: 72.52 ops/second - unlinked 0 (time 1776192101 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 796 (time 1776192200.92 total 10.00 last 79.58) - open/close 1516 (time 1776192210.92 total 20.01 last 71.98) - open/close 2257 (time 1776192220.94 total 30.02 last 73.96) - open/close 2934 (time 1776192230.94 total 40.03 last 67.68) - open/close 3658 (time 1776192240.94 total 50.03 last 72.39) - open/close 4410 (time 1776192250.94 total 60.03 last 75.20) - open/close 5180 (time 1776192260.95 total 70.04 last 76.91) - open/close 5940 (time 1776192270.96 total 80.05 last 75.94) - open/close 6687 (time 1776192280.97 total 90.05 last 74.66) - open/close 7423 (time 1776192290.97 total 100.06 last 73.58) - open/close 8220 (time 1776192300.97 total 110.06 last 79.69) - open/close 8926 (time 1776192310.98 total 120.06 last 70.56) - open/close 9531 (time 1776192320.98 total 130.06 last 60.49) total: 10000 open/close in 137.46 seconds: 72.75 ops/second - unlinked 0 (time 1776192332 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 527 (time 1776192423.42 total 10.00 last 52.68) - open/close 1331 (time 1776192433.43 total 20.01 last 80.35) - open/close 1938 (time 1776192443.43 total 30.01 last 60.68) - open/close 2626 (time 1776192453.44 total 40.02 last 68.71) - open/close 3332 (time 1776192463.45 total 50.03 last 70.55) - open/close 4108 (time 1776192473.46 total 60.04 last 77.53) - open/close 4880 (time 1776192483.46 total 70.04 last 77.20) - open/close 5571 (time 1776192493.46 total 80.04 last 69.10) - open/close 6345 (time 1776192503.47 total 90.05 last 77.36) - open/close 7155 (time 1776192513.48 total 100.06 last 80.91) - open/close 7940 (time 1776192523.48 total 110.06 last 78.47) - open/close 8679 (time 1776192533.49 total 120.07 last 73.84) - open/close 9483 (time 1776192543.49 total 130.08 last 80.35) total: 10000 open/close in 137.26 seconds: 72.86 ops/second - unlinked 0 (time 1776192555 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4196 82396 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253784 s, 4.1 MB/s fail_loc=0x80000204 Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server reboot facets: ost1 Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 2.08 seconds: 48.01 ops/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg435-server: LNET ready to unload oleg435-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.