== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:54:39 (1788872079) start mds service on oleg349-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1788872151.69 total 10.55 last 23.13) - open/close 493 (time 1788872161.87 total 20.72 last 24.47) - open/close 795 (time 1788872171.87 total 30.73 last 30.19) - open/close 1040 (time 1788872182.86 total 41.72 last 22.29) - open/close 1286 (time 1788872193.74 total 52.60 last 22.61) - open/close 1574 (time 1788872204.76 total 63.61 last 26.15) - open/close 1716 (time 1788872214.92 total 73.77 last 13.98) - open/close 1971 (time 1788872224.93 total 83.78 last 25.47) - open/close 2197 (time 1788872234.93 total 93.78 last 22.60) - open/close 2471 (time 1788872245.76 total 104.61 last 25.29) - open/close 2729 (time 1788872256.01 total 114.86 last 25.18) - open/close 2990 (time 1788872266.25 total 125.11 last 25.48) - open/close 3257 (time 1788872277.06 total 135.91 last 24.71) - open/close 3566 (time 1788872287.24 total 146.10 last 30.33) - open/close 3836 (time 1788872297.69 total 156.54 last 25.85) - open/close 4120 (time 1788872307.70 total 166.56 last 28.36) - open/close 4299 (time 1788872317.77 total 176.62 last 17.78) - open/close 4640 (time 1788872327.77 total 186.63 last 34.10) - open/close 4811 (time 1788872338.12 total 196.97 last 16.53) - open/close 5091 (time 1788872348.13 total 206.98 last 27.96) - open/close 5374 (time 1788872358.14 total 217.00 last 28.26) - open/close 5626 (time 1788872368.93 total 227.79 last 23.35) - open/close 5918 (time 1788872378.95 total 237.81 last 29.14) - open/close 6152 (time 1788872389.06 total 247.91 last 23.16) - open/close 6331 (time 1788872399.51 total 258.37 last 17.12) - open/close 6613 (time 1788872409.52 total 268.37 last 28.19) - open/close 6870 (time 1788872419.93 total 278.79 last 24.67) - open/close 7152 (time 1788872429.97 total 288.83 last 28.09) - open/close 7378 (time 1788872439.98 total 298.83 last 22.59) - open/close 7662 (time 1788872449.99 total 308.85 last 28.37) - open/close 7908 (time 1788872460.42 total 319.28 last 23.58) - open/close 8168 (time 1788872470.45 total 329.30 last 25.94) - open/close 8371 (time 1788872480.47 total 339.32 last 20.26) - open/close 8639 (time 1788872490.48 total 349.33 last 26.76) - open/close 8898 (time 1788872501.13 total 359.99 last 24.32) - open/close 9139 (time 1788872511.14 total 370.00 last 24.07) - open/close 9311 (time 1788872521.67 total 380.53 last 16.34) - open/close 9513 (time 1788872531.92 total 390.77 last 19.71) - open/close 9676 (time 1788872542.62 total 401.48 last 15.23) - open/close 9853 (time 1788872552.65 total 411.50 last 17.66) - open/close 10000 (time 1788872558.65 total 417.50 last 24.51) - open/close 10245 (time 1788872570.78 total 429.63 last 20.19) - open/close 10360 (time 1788872581.64 total 440.50 last 10.59) - open/close 10476 (time 1788872592.52 total 451.37 last 10.67) - open/close 10699 (time 1788872602.95 total 461.80 last 21.38) - open/close 10827 (time 1788872613.34 total 472.19 last 12.32) - open/close 11129 (time 1788872624.41 total 483.26 last 27.29) - open/close 11382 (time 1788872634.44 total 493.29 last 25.22) - open/close 11508 (time 1788872644.44 total 503.30 last 12.59) - open/close 11658 (time 1788872654.72 total 513.57 last 14.60) - open/close 11865 (time 1788872664.72 total 523.58 last 20.69) - open/close 12120 (time 1788872675.58 total 534.44 last 23.48) - open/close 12396 (time 1788872685.59 total 544.44 last 27.60) - open/close 12547 (time 1788872695.59 total 554.45 last 15.09) - open/close 12789 (time 1788872705.61 total 564.46 last 24.16) - open/close 12921 (time 1788872715.62 total 574.47 last 13.19) - open/close 13146 (time 1788872725.77 total 584.63 last 22.16) - open/close 13407 (time 1788872735.80 total 594.65 last 26.04) - open/close 13639 (time 1788872746.31 total 605.16 last 22.06) - open/close 13775 (time 1788872756.34 total 615.19 last 13.56) - open/close 13857 (time 1788872766.34 total 625.20 last 8.20) - open/close 14148 (time 1788872777.39 total 636.24 last 26.34) - open/close 14391 (time 1788872787.87 total 646.72 last 23.19) - open/close 14558 (time 1788872799.95 total 658.80 last 13.82) - open/close 14775 (time 1788872810.20 total 669.05 last 21.16) - open/close 15034 (time 1788872820.90 total 679.76 last 24.20) - open/close 15281 (time 1788872830.93 total 689.78 last 24.63) - open/close 15546 (time 1788872842.00 total 700.85 last 23.95) - open/close 15749 (time 1788872852.31 total 711.16 last 19.69) - open/close 15964 (time 1788872863.78 total 722.63 last 18.75) - open/close 16219 (time 1788872874.64 total 733.50 last 23.47) - open/close 16428 (time 1788872884.90 total 743.76 last 20.37) - open/close 16585 (time 1788872894.92 total 753.78 last 15.67) - open/close 16850 (time 1788872904.97 total 763.82 last 26.38) - open/close 17111 (time 1788872915.15 total 774.00 last 25.64) - open/close 17410 (time 1788872925.52 total 784.37 last 28.83) - open/close 17665 (time 1788872935.62 total 794.47 last 25.26) - open/close 17849 (time 1788872945.86 total 804.71 last 17.96) - open/close 18096 (time 1788872955.88 total 814.74 last 24.64) - open/close 18346 (time 1788872966.36 total 825.21 last 23.87) - open/close 18594 (time 1788872976.58 total 835.44 last 24.25) - open/close 18878 (time 1788872986.62 total 845.47 last 28.30) - open/close 19163 (time 1788872996.62 total 855.48 last 28.48) - open/close 19404 (time 1788873007.11 total 865.96 last 22.98) - open/close 19674 (time 1788873017.11 total 875.97 last 26.99) - open/close 19929 (time 1788873027.13 total 885.98 last 25.46) - open/close 20000 (time 1788873030.65 total 889.51 last 20.14) - open/close 20208 (time 1788873042.77 total 901.62 last 17.17) - open/close 20359 (time 1788873052.78 total 911.63 last 15.09) - open/close 20584 (time 1788873063.77 total 922.62 last 20.47) - open/close 20786 (time 1788873074.71 total 933.56 last 18.46) - open/close 21538 (time 1788873084.71 total 943.56 last 75.20) - open/close 22380 (time 1788873094.71 total 953.57 last 84.18) - open/close 23138 (time 1788873104.72 total 963.58 last 75.70) - open/close 23864 (time 1788873114.73 total 973.58 last 72.56) - open/close 24691 (time 1788873124.73 total 983.59 last 82.67) - open/close 25479 (time 1788873134.74 total 993.59 last 78.76) - open/close 26273 (time 1788873144.74 total 1003.59 last 79.39) - open/close 27051 (time 1788873154.76 total 1013.62 last 77.62) - open/close 27850 (time 1788873164.77 total 1023.62 last 79.86) - open/close 28621 (time 1788873174.77 total 1033.62 last 77.10) - open/close 29387 (time 1788873184.77 total 1043.62 last 76.60) - open/close 30000 (time 1788873192.41 total 1051.27 last 80.22) - open/close 30763 (time 1788873202.41 total 1061.27 last 76.27) - open/close 31544 (time 1788873212.42 total 1071.27 last 78.08) - open/close 32330 (time 1788873222.43 total 1081.28 last 78.52) - open/close 33157 (time 1788873232.43 total 1091.28 last 82.70) - open/close 33910 (time 1788873242.43 total 1101.28 last 75.29) - open/close 34689 (time 1788873252.44 total 1111.29 last 77.82) - open/close 35430 (time 1788873262.44 total 1121.29 last 74.09) - open/close 36048 (time 1788873272.45 total 1131.30 last 61.73) - open/close 36831 (time 1788873282.45 total 1141.31 last 78.28) - open/close 37542 (time 1788873292.48 total 1151.33 last 70.89) - open/close 38285 (time 1788873302.49 total 1161.34 last 74.24) - open/close 39037 (time 1788873312.50 total 1171.36 last 75.11) - open/close 39807 (time 1788873322.51 total 1181.37 last 76.92) - open/close 40000 (time 1788873325.15 total 1184.00 last 73.16) - open/close 40772 (time 1788873335.17 total 1194.02 last 77.08) - open/close 41519 (time 1788873345.17 total 1204.02 last 74.67) - open/close 42263 (time 1788873355.17 total 1214.02 last 74.40) - open/close 43041 (time 1788873365.17 total 1224.03 last 77.77) - open/close 43831 (time 1788873375.18 total 1234.04 last 78.94) - open/close 44560 (time 1788873385.19 total 1244.04 last 72.87) - open/close 45319 (time 1788873395.19 total 1254.04 last 75.87) - open/close 46036 (time 1788873405.20 total 1264.05 last 71.65) - open/close 46803 (time 1788873415.20 total 1274.05 last 76.69) - open/close 47554 (time 1788873425.21 total 1284.07 last 74.99) - open/close 48310 (time 1788873435.22 total 1294.08 last 75.52) - open/close 49004 (time 1788873445.23 total 1304.08 last 69.39) - open/close 49767 (time 1788873455.23 total 1314.08 last 76.29) total: 50000 open/close in 1316.85 seconds: 37.97 ops/second - unlinked 0 (time 1788873462 ; total 0 ; last 0) - unlinked 10000 (time 1788873600 ; total 138 ; last 138) - unlinked 20000 (time 1788873739 ; total 277 ; last 139) - unlinked 30000 (time 1788873878 ; total 416 ; last 139) - unlinked 40000 (time 1788874016 ; total 554 ; last 138) total: 50000 unlinks in 692 seconds: 72.254333 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2