== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:53:50 (1781229230) start mds service on oleg208-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg208-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1781229324.60 total 10.11 last 12.95) - open/close 312 (time 1781229334.61 total 20.12 last 18.08) - open/close 493 (time 1781229345.26 total 30.77 last 17.00) - open/close 757 (time 1781229356.10 total 41.61 last 24.35) - open/close 1015 (time 1781229366.61 total 52.12 last 24.56) - open/close 1170 (time 1781229376.65 total 62.16 last 15.44) - open/close 1295 (time 1781229386.74 total 72.25 last 12.39) - open/close 1397 (time 1781229397.63 total 83.14 last 9.37) - open/close 1592 (time 1781229410.33 total 95.84 last 15.36) - open/close 1818 (time 1781229420.37 total 105.88 last 22.51) - open/close 2004 (time 1781229430.41 total 115.92 last 18.51) - open/close 2179 (time 1781229440.63 total 126.14 last 17.12) - open/close 2386 (time 1781229450.64 total 136.15 last 20.69) - open/close 2599 (time 1781229461.17 total 146.68 last 20.22) - open/close 2862 (time 1781229471.66 total 157.17 last 25.08) - open/close 3035 (time 1781229482.25 total 167.76 last 16.34) - open/close 3266 (time 1781229492.56 total 178.07 last 22.40) - open/close 3493 (time 1781229504.64 total 190.15 last 18.80) - open/close 3620 (time 1781229514.68 total 200.19 last 12.65) - open/close 3793 (time 1781229524.68 total 210.19 last 17.29) - open/close 4032 (time 1781229534.71 total 220.22 last 23.83) - open/close 4194 (time 1781229545.59 total 231.10 last 14.89) - open/close 4392 (time 1781229558.91 total 244.42 last 14.86) - open/close 4622 (time 1781229569.92 total 255.43 last 20.90) - open/close 4792 (time 1781229579.92 total 265.43 last 16.98) - open/close 4909 (time 1781229590.72 total 276.23 last 10.84) - open/close 5013 (time 1781229601.52 total 287.03 last 9.63) - open/close 5130 (time 1781229611.58 total 297.09 last 11.62) - open/close 5229 (time 1781229621.59 total 307.10 last 9.89) - open/close 5355 (time 1781229631.62 total 317.13 last 12.56) - open/close 5554 (time 1781229641.64 total 327.15 last 19.86) - open/close 5677 (time 1781229652.08 total 337.59 last 11.79) - open/close 5866 (time 1781229662.08 total 347.59 last 18.89) - open/close 5988 (time 1781229672.95 total 358.46 last 11.22) - open/close 6180 (time 1781229682.95 total 368.46 last 19.19) - open/close 6379 (time 1781229693.94 total 379.45 last 18.11) - open/close 6522 (time 1781229703.95 total 389.46 last 14.28) - open/close 6620 (time 1781229715.65 total 401.16 last 8.38) - open/close 6780 (time 1781229725.66 total 411.17 last 15.99) - open/close 6934 (time 1781229737.33 total 422.84 last 13.19) - open/close 7144 (time 1781229747.52 total 433.03 last 20.62) - open/close 7323 (time 1781229757.52 total 443.03 last 17.89) - open/close 7450 (time 1781229767.68 total 453.19 last 12.50) - open/close 7628 (time 1781229778.61 total 464.12 last 16.29) - open/close 7774 (time 1781229789.11 total 474.62 last 13.90) - open/close 7978 (time 1781229799.82 total 485.33 last 19.05) - open/close 8083 (time 1781229809.85 total 495.36 last 10.47) - open/close 8203 (time 1781229819.87 total 505.38 last 11.98) - open/close 8299 (time 1781229830.64 total 516.15 last 8.91) - open/close 8504 (time 1781229840.81 total 526.32 last 20.15) - open/close 8563 (time 1781229850.83 total 536.34 last 5.89) - open/close 8714 (time 1781229861.77 total 547.28 last 13.80) - open/close 8920 (time 1781229871.99 total 557.50 last 20.16) - open/close 9117 (time 1781229882.94 total 568.45 last 17.99) - open/close 9263 (time 1781229893.18 total 578.69 last 14.25) - open/close 9409 (time 1781229903.20 total 588.71 last 14.57) - open/close 9580 (time 1781229913.65 total 599.16 last 16.37) - open/close 9730 (time 1781229923.78 total 609.29 last 14.81) - open/close 9856 (time 1781229933.81 total 619.32 last 12.56) - open/close 9981 (time 1781229944.06 total 629.57 last 12.19) - open/close 10145 (time 1781229954.07 total 639.58 last 16.38) - open/close 10320 (time 1781229964.10 total 649.61 last 17.44) - open/close 10471 (time 1781229974.55 total 660.06 last 14.46) - open/close 10706 (time 1781229984.70 total 670.21 last 23.14) - open/close 10877 (time 1781229994.95 total 680.46 last 16.69) - open/close 10989 (time 1781230005.63 total 691.14 last 10.49) - open/close 11150 (time 1781230016.70 total 702.21 last 14.54) - open/close 11366 (time 1781230027.56 total 713.07 last 19.88) - open/close 11633 (time 1781230040.45 total 725.96 last 20.72) - open/close 11859 (time 1781230050.45 total 735.96 last 22.60) - open/close 12047 (time 1781230061.56 total 747.07 last 16.92) - open/close 12167 (time 1781230071.57 total 757.08 last 12.00) - open/close 12345 (time 1781230086.47 total 771.98 last 11.94) - open/close 12410 (time 1781230096.50 total 782.01 last 6.48) - open/close 12463 (time 1781230106.51 total 792.02 last 5.29) - open/close 12649 (time 1781230116.53 total 802.05 last 18.56) - open/close 12812 (time 1781230127.43 total 812.94 last 14.96) - open/close 12889 (time 1781230137.73 total 823.24 last 7.48) - open/close 13022 (time 1781230148.29 total 833.80 last 12.60) - open/close 13185 (time 1781230158.29 total 843.80 last 16.29) - open/close 13356 (time 1781230168.96 total 854.47 last 16.03) - open/close 13649 (time 1781230178.96 total 864.47 last 29.30) - open/close 13829 (time 1781230189.01 total 874.52 last 17.92) - open/close 13967 (time 1781230199.30 total 884.81 last 13.41) - open/close 14028 (time 1781230209.42 total 894.93 last 6.03) - open/close 14125 (time 1781230222.19 total 907.70 last 7.59) - open/close 14267 (time 1781230232.22 total 917.73 last 14.16) - open/close 14379 (time 1781230242.77 total 928.28 last 10.61) - open/close 14515 (time 1781230253.49 total 939.00 last 12.69) - open/close 14614 (time 1781230263.49 total 949.01 last 9.90) - open/close 14758 (time 1781230274.35 total 959.86 last 13.26) - open/close 14985 (time 1781230284.99 total 970.50 last 21.35) - open/close 15146 (time 1781230295.05 total 980.56 last 16.00) - open/close 15357 (time 1781230305.06 total 990.57 last 21.08) - open/close 15513 (time 1781230315.07 total 1000.58 last 15.58) - open/close 15690 (time 1781230325.09 total 1010.60 last 17.67) - open/close 15789 (time 1781230335.66 total 1021.17 last 9.37) - open/close 15976 (time 1781230345.73 total 1031.24 last 18.57) - open/close 16203 (time 1781230355.76 total 1041.27 last 22.65) - open/close 16418 (time 1781230368.65 total 1054.16 last 16.67) - open/close 16582 (time 1781230378.66 total 1064.17 last 16.39) - open/close 17192 (time 1781230388.67 total 1074.18 last 60.95) - open/close 17931 (time 1781230398.67 total 1084.18 last 73.86) - open/close 18695 (time 1781230408.68 total 1094.20 last 76.32) - open/close 19375 (time 1781230418.69 total 1104.20 last 67.94) - open/close 20000 (time 1781230428.07 total 1113.58 last 66.68) - open/close 20701 (time 1781230438.07 total 1123.58 last 70.05) - open/close 21408 (time 1781230448.12 total 1133.63 last 70.40) - open/close 22094 (time 1781230458.13 total 1143.64 last 68.52) - open/close 22746 (time 1781230468.13 total 1153.64 last 65.20) - open/close 23436 (time 1781230478.13 total 1163.64 last 68.96) - open/close 24183 (time 1781230488.15 total 1173.66 last 74.60) - open/close 24905 (time 1781230498.15 total 1183.66 last 72.18) - open/close 25638 (time 1781230508.16 total 1193.67 last 73.24) - open/close 26269 (time 1781230518.16 total 1203.67 last 63.09) - open/close 26989 (time 1781230528.17 total 1213.68 last 71.96) - open/close 27589 (time 1781230538.19 total 1223.70 last 59.86) - open/close 28174 (time 1781230548.19 total 1233.70 last 58.49) - open/close 28900 (time 1781230558.21 total 1243.72 last 72.48) - open/close 29568 (time 1781230568.21 total 1253.73 last 66.76) - open/close 30000 (time 1781230574.87 total 1260.38 last 64.89) - open/close 30664 (time 1781230584.88 total 1270.39 last 66.35) - open/close 31299 (time 1781230594.89 total 1280.40 last 63.44) - open/close 32076 (time 1781230604.90 total 1290.41 last 77.63) - open/close 32846 (time 1781230614.90 total 1300.41 last 76.98) - open/close 33557 (time 1781230624.92 total 1310.43 last 70.99) - open/close 34236 (time 1781230634.93 total 1320.44 last 67.82) - open/close 34992 (time 1781230644.93 total 1330.44 last 75.58) - open/close 35707 (time 1781230654.94 total 1340.45 last 71.46) - open/close 36339 (time 1781230664.94 total 1350.45 last 63.18) - open/close 37011 (time 1781230674.96 total 1360.47 last 67.08) - open/close 37660 (time 1781230684.97 total 1370.48 last 64.84) - open/close 38332 (time 1781230694.97 total 1380.49 last 67.15) - open/close 38985 (time 1781230704.98 total 1390.49 last 65.29) - open/close 39702 (time 1781230715.03 total 1400.54 last 71.34) - open/close 40000 (time 1781230718.60 total 1404.11 last 83.50) - open/close 40712 (time 1781230728.60 total 1414.11 last 71.17) - open/close 41445 (time 1781230738.61 total 1424.12 last 73.24) - open/close 42163 (time 1781230748.61 total 1434.12 last 71.78) - open/close 42755 (time 1781230758.62 total 1444.13 last 59.14) - open/close 43303 (time 1781230768.63 total 1454.14 last 54.77) - open/close 43985 (time 1781230778.64 total 1464.15 last 68.10) - open/close 44533 (time 1781230788.67 total 1474.18 last 54.65) - open/close 45136 (time 1781230798.70 total 1484.21 last 60.13) - open/close 45731 (time 1781230808.71 total 1494.22 last 59.45) - open/close 46290 (time 1781230818.71 total 1504.22 last 55.86) - open/close 47002 (time 1781230828.73 total 1514.24 last 71.10) - open/close 47631 (time 1781230838.73 total 1524.24 last 62.86) - open/close 48331 (time 1781230848.75 total 1534.26 last 69.89) - open/close 48927 (time 1781230858.80 total 1544.31 last 59.32) - open/close 49629 (time 1781230868.81 total 1554.32 last 70.12) total: 50000 open/close in 1560.33 seconds: 32.04 ops/second - unlinked 0 (time 1781230879 ; total 0 ; last 0) - unlinked 10000 (time 1781231033 ; total 154 ; last 154) - unlinked 20000 (time 1781231184 ; total 305 ; last 151) - unlinked 30000 (time 1781231346 ; total 467 ; last 162) - unlinked 40000 (time 1781231511 ; total 632 ; last 165) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: modules unloaded. pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2