== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:23:32 (1782246212) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1782246333.08 total 10.39 last 23.01) - open/close 462 (time 1782246343.09 total 20.39 last 22.30) - open/close 681 (time 1782246355.64 total 32.94 last 17.45) - open/close 730 (time 1782246365.68 total 42.98 last 4.88) - open/close 949 (time 1782246375.69 total 52.99 last 21.88) - open/close 1133 (time 1782246385.75 total 63.05 last 18.28) - open/close 1320 (time 1782246396.47 total 73.77 last 17.45) - open/close 1449 (time 1782246406.69 total 83.99 last 12.62) - open/close 1711 (time 1782246416.69 total 94.00 last 26.19) - open/close 1922 (time 1782246426.78 total 104.08 last 20.92) - open/close 2183 (time 1782246437.66 total 114.96 last 23.98) - open/close 2410 (time 1782246448.29 total 125.60 last 21.35) - open/close 2567 (time 1782246458.30 total 135.60 last 15.70) - open/close 2678 (time 1782246469.94 total 147.25 last 9.53) - open/close 2839 (time 1782246480.44 total 157.74 last 15.34) - open/close 3019 (time 1782246490.44 total 167.74 last 17.99) - open/close 3273 (time 1782246500.91 total 178.21 last 24.25) - open/close 3504 (time 1782246511.25 total 188.56 last 22.33) - open/close 3714 (time 1782246521.30 total 198.60 last 20.90) - open/close 3968 (time 1782246531.32 total 208.63 last 25.34) - open/close 4195 (time 1782246541.98 total 219.28 last 21.30) - open/close 4304 (time 1782246552.57 total 229.87 last 10.30) - open/close 4474 (time 1782246563.11 total 240.42 last 16.12) - open/close 4650 (time 1782246573.13 total 250.43 last 17.57) - open/close 4793 (time 1782246583.13 total 260.43 last 14.30) - open/close 4848 (time 1782246594.91 total 272.21 last 4.67) - open/close 4986 (time 1782246607.00 total 284.30 last 11.41) - open/close 5186 (time 1782246617.05 total 294.36 last 19.90) - open/close 5277 (time 1782246628.07 total 305.38 last 8.26) - open/close 5524 (time 1782246638.09 total 315.40 last 24.65) - open/close 5727 (time 1782246648.10 total 325.40 last 20.29) - open/close 5939 (time 1782246658.41 total 335.72 last 20.55) - open/close 6220 (time 1782246668.42 total 345.73 last 28.07) - open/close 6418 (time 1782246678.78 total 356.08 last 19.13) - open/close 6538 (time 1782246688.79 total 366.09 last 11.98) - open/close 6617 (time 1782246699.17 total 376.47 last 7.61) - open/close 6789 (time 1782246709.42 total 386.72 last 16.78) - open/close 6902 (time 1782246720.42 total 397.72 last 10.27) - open/close 7093 (time 1782246730.42 total 407.72 last 19.09) - open/close 7325 (time 1782246740.79 total 418.10 last 22.37) - open/close 7517 (time 1782246752.48 total 429.79 last 16.42) - open/close 7652 (time 1782246762.49 total 439.79 last 13.49) - open/close 7840 (time 1782246772.50 total 449.80 last 18.78) - open/close 7977 (time 1782246782.84 total 460.14 last 13.25) - open/close 8082 (time 1782246793.18 total 470.49 last 10.15) - open/close 8327 (time 1782246803.63 total 480.93 last 23.46) - open/close 8543 (time 1782246814.62 total 491.93 last 19.64) - open/close 8799 (time 1782246824.63 total 501.93 last 25.58) - open/close 8999 (time 1782246834.64 total 511.94 last 19.98) - open/close 9168 (time 1782246844.65 total 521.95 last 16.89) - open/close 9286 (time 1782246855.55 total 532.85 last 10.82) - open/close 9463 (time 1782246866.35 total 543.65 last 16.39) - open/close 9549 (time 1782246876.75 total 554.05 last 8.27) - open/close 9661 (time 1782246886.95 total 564.25 last 10.98) - open/close 9744 (time 1782246897.96 total 575.26 last 7.54) - open/close 9826 (time 1782246908.83 total 586.13 last 7.54) - open/close 9977 (time 1782246919.27 total 596.57 last 14.47) - open/close 10146 (time 1782246932.03 total 609.33 last 13.24) - open/close 10315 (time 1782246942.95 total 620.25 last 15.48) - open/close 10432 (time 1782246953.05 total 630.36 last 11.58) - open/close 10651 (time 1782246963.06 total 640.36 last 21.89) - open/close 10823 (time 1782246973.06 total 650.36 last 17.20) - open/close 10999 (time 1782246985.77 total 663.07 last 13.85) - open/close 11192 (time 1782246996.64 total 673.95 last 17.75) - open/close 11274 (time 1782247006.64 total 683.95 last 8.20) - open/close 11497 (time 1782247016.99 total 694.29 last 21.55) - open/close 11706 (time 1782247027.05 total 704.35 last 20.78) - open/close 11881 (time 1782247038.02 total 715.32 last 15.95) - open/close 12024 (time 1782247048.43 total 725.73 last 13.73) - open/close 12283 (time 1782247058.44 total 735.75 last 25.87) - open/close 12494 (time 1782247068.70 total 746.00 last 20.57) - open/close 12681 (time 1782247078.71 total 756.01 last 18.68) - open/close 12777 (time 1782247089.07 total 766.37 last 9.26) - open/close 13092 (time 1782247100.08 total 777.38 last 28.62) - open/close 13178 (time 1782247110.50 total 787.80 last 8.25) - open/close 13381 (time 1782247120.50 total 797.81 last 20.29) - open/close 13529 (time 1782247130.91 total 808.21 last 14.22) - open/close 13773 (time 1782247141.72 total 819.02 last 22.57) - open/close 14005 (time 1782247152.19 total 829.49 last 22.17) - open/close 14163 (time 1782247162.62 total 839.92 last 15.15) - open/close 14433 (time 1782247172.62 total 849.92 last 26.99) - open/close 14641 (time 1782247184.74 total 862.04 last 17.16) - open/close 14830 (time 1782247195.55 total 872.85 last 17.49) - open/close 15059 (time 1782247205.79 total 883.09 last 22.36) - open/close 15330 (time 1782247216.74 total 894.04 last 24.75) - open/close 15468 (time 1782247226.74 total 904.05 last 13.79) - open/close 15647 (time 1782247236.76 total 914.06 last 17.87) - open/close 15791 (time 1782247246.90 total 924.21 last 14.19) - open/close 16020 (time 1782247257.82 total 935.12 last 20.98) - open/close 16211 (time 1782247268.51 total 945.81 last 17.87) - open/close 16345 (time 1782247279.17 total 956.47 last 12.57) - open/close 16585 (time 1782247289.69 total 967.00 last 22.81) - open/close 16778 (time 1782247299.71 total 977.01 last 19.27) - open/close 17021 (time 1782247309.90 total 987.21 last 23.83) - open/close 17142 (time 1782247322.61 total 999.91 last 9.53) - open/close 17292 (time 1782247332.62 total 1009.92 last 14.98) - open/close 17431 (time 1782247342.69 total 1020.00 last 13.80) - open/close 17703 (time 1782247352.72 total 1030.02 last 27.14) - open/close 18417 (time 1782247362.72 total 1040.02 last 71.35) - open/close 19162 (time 1782247372.72 total 1050.03 last 74.48) - open/close 19820 (time 1782247382.73 total 1060.04 last 65.73) - open/close 20000 (time 1782247385.61 total 1062.91 last 62.62) - open/close 20646 (time 1782247395.61 total 1072.91 last 64.58) - open/close 21317 (time 1782247405.62 total 1082.92 last 67.05) - open/close 22002 (time 1782247415.63 total 1092.93 last 68.42) - open/close 22702 (time 1782247425.67 total 1102.97 last 69.75) - open/close 23301 (time 1782247435.67 total 1112.97 last 59.89) - open/close 23879 (time 1782247445.68 total 1122.99 last 57.71) - open/close 24409 (time 1782247455.69 total 1132.99 last 52.97) - open/close 25145 (time 1782247465.69 total 1143.00 last 73.59) - open/close 25700 (time 1782247475.69 total 1153.00 last 55.50) - open/close 26321 (time 1782247485.70 total 1163.00 last 62.08) - open/close 26908 (time 1782247495.73 total 1173.03 last 58.52) - open/close 27545 (time 1782247505.73 total 1183.04 last 63.65) - open/close 28243 (time 1782247515.74 total 1193.04 last 69.75) - open/close 28928 (time 1782247525.75 total 1203.05 last 68.45) - open/close 29591 (time 1782247535.75 total 1213.06 last 66.27) - open/close 30000 (time 1782247541.69 total 1218.99 last 68.87) - open/close 30659 (time 1782247551.70 total 1229.00 last 65.87) - open/close 31346 (time 1782247561.70 total 1239.00 last 68.69) - open/close 32019 (time 1782247571.71 total 1249.01 last 67.22) - open/close 32685 (time 1782247581.72 total 1259.02 last 66.55) - open/close 33364 (time 1782247591.72 total 1269.03 last 67.86) - open/close 34054 (time 1782247601.72 total 1279.03 last 69.00) - open/close 34737 (time 1782247611.73 total 1289.03 last 68.27) - open/close 35433 (time 1782247621.73 total 1299.03 last 69.60) - open/close 36071 (time 1782247631.73 total 1309.03 last 63.78) - open/close 36725 (time 1782247641.74 total 1319.04 last 65.35) - open/close 37413 (time 1782247651.76 total 1329.06 last 68.70) - open/close 38031 (time 1782247661.79 total 1339.09 last 61.57) - open/close 38696 (time 1782247671.80 total 1349.11 last 66.43) - open/close 39279 (time 1782247681.81 total 1359.11 last 58.27) - open/close 39874 (time 1782247691.82 total 1369.12 last 59.44) - open/close 40000 (time 1782247694.12 total 1371.43 last 54.61) - open/close 40570 (time 1782247704.13 total 1381.43 last 56.98) - open/close 41212 (time 1782247714.13 total 1391.43 last 64.18) - open/close 41882 (time 1782247724.13 total 1401.43 last 67.00) - open/close 42543 (time 1782247734.14 total 1411.44 last 66.06) - open/close 43188 (time 1782247744.16 total 1421.47 last 64.33) - open/close 43883 (time 1782247754.17 total 1431.47 last 69.45) - open/close 44553 (time 1782247764.18 total 1441.48 last 66.94) - open/close 45209 (time 1782247774.19 total 1451.50 last 65.50) - open/close 45858 (time 1782247784.20 total 1461.50 last 64.88) - open/close 46484 (time 1782247794.20 total 1471.50 last 62.60) - open/close 47067 (time 1782247804.20 total 1481.50 last 58.28) - open/close 47710 (time 1782247814.20 total 1491.51 last 64.29) - open/close 48325 (time 1782247824.21 total 1501.52 last 61.43) - open/close 49051 (time 1782247834.22 total 1511.52 last 72.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1520.83 seconds: 32.63 ops/second - unlinked 0 (time 1782247848 ; total 0 ; last 0) - unlinked 10000 (time 1782248011 ; total 163 ; last 163) - unlinked 20000 (time 1782248170 ; total 322 ; last 159) - unlinked 30000 (time 1782248332 ; total 484 ; last 162) - unlinked 40000 (time 1782248489 ; total 641 ; last 157) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 786 seconds: 63.143764 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: modules unloaded. pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2