== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:07:36 (1787382456) start mds service on oleg335-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg335-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 211 (time 1787382566.46 total 10.16 last 20.77) - open/close 405 (time 1787382577.48 total 21.17 last 17.61) - open/close 688 (time 1787382587.48 total 31.18 last 28.29) - open/close 831 (time 1787382597.51 total 41.21 last 14.26) - open/close 1054 (time 1787382607.52 total 51.22 last 22.27) - open/close 1252 (time 1787382618.43 total 62.13 last 18.15) - open/close 1469 (time 1787382628.43 total 72.13 last 21.70) - open/close 1688 (time 1787382638.61 total 82.31 last 21.52) - open/close 1947 (time 1787382648.61 total 92.31 last 25.89) - open/close 2208 (time 1787382659.65 total 103.35 last 23.64) - open/close 2442 (time 1787382669.66 total 113.36 last 23.38) - open/close 2645 (time 1787382679.68 total 123.37 last 20.27) - open/close 2932 (time 1787382689.70 total 133.40 last 28.62) - open/close 3153 (time 1787382700.29 total 143.99 last 20.88) - open/close 3423 (time 1787382710.29 total 153.99 last 26.99) - open/close 3629 (time 1787382720.30 total 164.00 last 20.59) - open/close 3844 (time 1787382730.30 total 174.00 last 21.50) - open/close 4029 (time 1787382740.31 total 184.01 last 18.48) - open/close 4224 (time 1787382750.98 total 194.68 last 18.28) - open/close 4481 (time 1787382761.44 total 205.14 last 24.58) - open/close 4683 (time 1787382772.02 total 215.72 last 19.08) - open/close 4926 (time 1787382782.77 total 226.47 last 22.61) - open/close 5121 (time 1787382793.79 total 237.49 last 17.70) - open/close 5347 (time 1787382803.89 total 247.59 last 22.37) - open/close 5566 (time 1787382813.91 total 257.60 last 21.88) - open/close 5802 (time 1787382824.00 total 267.69 last 23.39) - open/close 6027 (time 1787382834.01 total 277.71 last 22.47) - open/close 6266 (time 1787382844.11 total 287.80 last 23.67) - open/close 6565 (time 1787382854.14 total 297.84 last 29.81) - open/close 6810 (time 1787382864.14 total 307.84 last 24.48) - open/close 7074 (time 1787382874.18 total 317.88 last 26.31) - open/close 7215 (time 1787382884.35 total 328.05 last 13.86) - open/close 7471 (time 1787382894.99 total 338.68 last 24.07) - open/close 7699 (time 1787382905.05 total 348.74 last 22.66) - open/close 7853 (time 1787382915.20 total 358.89 last 15.17) - open/close 8070 (time 1787382925.55 total 369.25 last 20.96) - open/close 8154 (time 1787382935.59 total 379.28 last 8.37) - open/close 8415 (time 1787382945.60 total 389.30 last 26.07) - open/close 8627 (time 1787382955.75 total 399.44 last 20.89) - open/close 8831 (time 1787382965.87 total 409.57 last 20.14) - open/close 9039 (time 1787382976.78 total 420.48 last 19.08) - open/close 9308 (time 1787382986.80 total 430.50 last 26.85) - open/close 9525 (time 1787382996.80 total 440.50 last 21.69) - open/close 9735 (time 1787383006.82 total 450.52 last 20.96) - open/close 9951 (time 1787383016.83 total 460.53 last 21.59) - open/close 10192 (time 1787383027.08 total 470.78 last 23.51) - open/close 10423 (time 1787383037.08 total 480.78 last 23.09) - open/close 10655 (time 1787383047.09 total 490.79 last 23.19) - open/close 10865 (time 1787383057.09 total 500.79 last 21.00) - open/close 10953 (time 1787383068.26 total 511.96 last 7.88) - open/close 11214 (time 1787383079.29 total 522.99 last 23.66) - open/close 11405 (time 1787383089.34 total 533.04 last 19.00) - open/close 11661 (time 1787383100.02 total 543.72 last 23.97) - open/close 11942 (time 1787383110.03 total 553.73 last 28.08) - open/close 12202 (time 1787383123.15 total 566.85 last 19.82) - open/close 12446 (time 1787383133.52 total 577.22 last 23.53) - open/close 12619 (time 1787383143.71 total 587.40 last 16.98) - open/close 12828 (time 1787383153.94 total 597.64 last 20.41) - open/close 13026 (time 1787383163.95 total 607.65 last 19.79) - open/close 13160 (time 1787383173.95 total 617.65 last 13.40) - open/close 13385 (time 1787383183.96 total 627.66 last 22.49) - open/close 13688 (time 1787383193.97 total 637.67 last 30.27) - open/close 13915 (time 1787383203.98 total 647.67 last 22.69) - open/close 14178 (time 1787383214.34 total 658.04 last 25.37) - open/close 14437 (time 1787383225.43 total 669.13 last 23.36) - open/close 14674 (time 1787383235.97 total 679.66 last 22.49) - open/close 14947 (time 1787383246.59 total 690.29 last 25.69) - open/close 15185 (time 1787383258.13 total 701.83 last 20.63) - open/close 15400 (time 1787383268.59 total 712.29 last 20.55) - open/close 15687 (time 1787383278.59 total 722.29 last 28.69) - open/close 15913 (time 1787383288.79 total 732.49 last 22.16) - open/close 16154 (time 1787383299.14 total 742.84 last 23.29) - open/close 16393 (time 1787383309.58 total 753.28 last 22.89) - open/close 16619 (time 1787383319.58 total 763.28 last 22.60) - open/close 16877 (time 1787383329.59 total 773.29 last 25.78) - open/close 17131 (time 1787383339.62 total 783.32 last 25.32) - open/close 17341 (time 1787383350.47 total 794.17 last 19.36) - open/close 17509 (time 1787383363.59 total 807.29 last 12.80) - open/close 17757 (time 1787383374.09 total 817.79 last 23.62) - open/close 17933 (time 1787383385.74 total 829.43 last 15.12) - open/close 18116 (time 1787383396.42 total 840.12 last 17.13) - open/close 18379 (time 1787383407.05 total 850.74 last 24.75) - open/close 18537 (time 1787383417.28 total 860.97 last 15.45) - open/close 18716 (time 1787383427.78 total 871.48 last 17.04) - open/close 18953 (time 1787383439.31 total 883.01 last 20.56) - open/close 19192 (time 1787383449.32 total 893.02 last 23.89) - open/close 19419 (time 1787383459.58 total 903.28 last 22.11) - open/close 19654 (time 1787383469.64 total 913.34 last 23.36) - open/close 20000 (time 1787383477.09 total 920.79 last 46.49) - open/close 20766 (time 1787383487.09 total 930.79 last 76.58) - open/close 21523 (time 1787383497.11 total 940.81 last 75.57) - open/close 22234 (time 1787383507.11 total 950.81 last 71.10) - open/close 22983 (time 1787383517.11 total 960.81 last 74.86) - open/close 23701 (time 1787383527.11 total 970.81 last 71.79) - open/close 24423 (time 1787383537.13 total 980.83 last 72.10) - open/close 25073 (time 1787383547.14 total 990.84 last 64.90) - open/close 25574 (time 1787383557.16 total 1000.85 last 50.03) - open/close 26157 (time 1787383567.16 total 1010.86 last 58.26) - open/close 26857 (time 1787383577.19 total 1020.89 last 69.83) - open/close 27586 (time 1787383587.19 total 1030.89 last 72.87) - open/close 28300 (time 1787383597.21 total 1040.91 last 71.29) - open/close 29003 (time 1787383607.21 total 1050.91 last 70.27) - open/close 29732 (time 1787383617.22 total 1060.91 last 72.88) - open/close 30000 (time 1787383620.97 total 1064.67 last 71.38) - open/close 30695 (time 1787383631.00 total 1074.70 last 69.31) - open/close 31409 (time 1787383641.01 total 1084.71 last 71.32) - open/close 32173 (time 1787383651.01 total 1094.71 last 76.37) - open/close 32857 (time 1787383661.02 total 1104.72 last 68.35) - open/close 33540 (time 1787383671.03 total 1114.73 last 68.24) - open/close 34087 (time 1787383681.04 total 1124.73 last 54.67) - open/close 34673 (time 1787383691.05 total 1134.75 last 58.53) - open/close 35369 (time 1787383701.05 total 1144.75 last 69.58) - open/close 36133 (time 1787383711.06 total 1154.76 last 76.30) - open/close 36870 (time 1787383721.07 total 1164.76 last 73.68) - open/close 37573 (time 1787383731.09 total 1174.79 last 70.11) - open/close 38283 (time 1787383741.10 total 1184.80 last 70.94) - open/close 39027 (time 1787383751.11 total 1194.80 last 74.38) - open/close 39699 (time 1787383761.11 total 1204.81 last 67.14) - open/close 40000 (time 1787383765.17 total 1208.87 last 74.23) - open/close 40677 (time 1787383775.17 total 1218.87 last 67.68) - open/close 41328 (time 1787383785.17 total 1228.87 last 65.09) - open/close 42032 (time 1787383795.17 total 1238.87 last 70.39) - open/close 42724 (time 1787383805.19 total 1248.89 last 69.11) - open/close 43323 (time 1787383815.21 total 1258.91 last 59.77) - open/close 43995 (time 1787383825.22 total 1268.92 last 67.15) - open/close 44612 (time 1787383835.23 total 1278.93 last 61.60) - open/close 45274 (time 1787383845.26 total 1288.95 last 66.05) - open/close 45887 (time 1787383855.26 total 1298.96 last 61.27) - open/close 46479 (time 1787383865.27 total 1308.97 last 59.14) - open/close 47045 (time 1787383875.27 total 1318.97 last 56.59) - open/close 47730 (time 1787383885.28 total 1328.97 last 68.48) - open/close 48412 (time 1787383895.29 total 1338.99 last 68.10) - open/close 49089 (time 1787383905.29 total 1348.99 last 67.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1356.87 seconds: 36.58 ops/second - unlinked 0 (time 1787383916 ; total 0 ; last 0) - unlinked 10000 (time 1787384059 ; total 143 ; last 143) - unlinked 20000 (time 1787384210 ; total 294 ; last 151) - unlinked 30000 (time 1787384360 ; total 444 ; last 150) - unlinked 40000 (time 1787384504 ; total 588 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 724 seconds: 68.551102 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2