== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:58:55 (1788314335) start mds service on oleg608-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing load_modules_local oleg608-server: Loading modules from /home/green/git/lustre-release/lustre oleg608-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg608-server: detected 4 online CPUs by sysfs oleg608-server: MODOPTS_LIBCFS= oleg608-server: Force libcfs to create 2 CPU partitions oleg608-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg608-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg608-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg608-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg608-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Started lustre-OST0000 oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg608-client.virtnet: -o user_xattr,flock 192.168.206.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 233 (time 1788314445.11 total 10.00 last 23.29) - open/close 475 (time 1788314455.14 total 20.03 last 24.14) - open/close 715 (time 1788314465.14 total 30.03 last 23.98) - open/close 970 (time 1788314475.16 total 40.05 last 25.45) - open/close 1199 (time 1788314485.16 total 50.05 last 22.90) - open/close 1449 (time 1788314495.18 total 60.07 last 24.96) - open/close 1702 (time 1788314505.31 total 70.20 last 24.96) - open/close 1979 (time 1788314515.62 total 80.51 last 26.87) - open/close 2224 (time 1788314526.52 total 91.41 last 22.49) - open/close 2470 (time 1788314537.32 total 102.21 last 22.76) - open/close 2737 (time 1788314547.33 total 112.22 last 26.68) - open/close 2948 (time 1788314558.00 total 122.89 last 19.78) - open/close 3154 (time 1788314568.30 total 133.19 last 20.00) - open/close 3386 (time 1788314578.30 total 143.19 last 23.19) - open/close 3616 (time 1788314588.31 total 153.20 last 22.97) - open/close 3841 (time 1788314598.32 total 163.21 last 22.48) - open/close 4094 (time 1788314608.33 total 173.22 last 25.27) - open/close 4286 (time 1788314618.34 total 183.23 last 19.19) - open/close 4520 (time 1788314628.35 total 193.24 last 23.37) - open/close 4822 (time 1788314639.21 total 204.10 last 27.81) - open/close 5123 (time 1788314649.96 total 214.85 last 27.99) - open/close 5427 (time 1788314659.96 total 224.85 last 30.40) - open/close 5645 (time 1788314670.08 total 234.97 last 21.56) - open/close 5939 (time 1788314680.09 total 244.98 last 29.35) - open/close 6201 (time 1788314690.09 total 254.98 last 26.20) - open/close 6437 (time 1788314700.10 total 264.99 last 23.60) - open/close 6677 (time 1788314710.88 total 275.77 last 22.25) - open/close 6922 (time 1788314720.89 total 285.78 last 24.49) - open/close 7107 (time 1788314731.80 total 296.69 last 16.95) - open/close 7334 (time 1788314741.86 total 306.75 last 22.57) - open/close 7536 (time 1788314752.29 total 317.18 last 19.38) - open/close 7800 (time 1788314762.30 total 327.19 last 26.37) - open/close 8027 (time 1788314772.30 total 337.19 last 22.70) - open/close 8262 (time 1788314782.31 total 347.20 last 23.48) - open/close 8528 (time 1788314792.68 total 357.57 last 25.63) - open/close 8829 (time 1788314803.55 total 368.44 last 27.69) - open/close 9054 (time 1788314813.56 total 378.45 last 22.49) - open/close 9262 (time 1788314824.12 total 389.01 last 19.69) - open/close 9524 (time 1788314834.84 total 399.73 last 24.44) - open/close 9783 (time 1788314844.85 total 409.74 last 25.88) - open/close 10000 (time 1788314854.04 total 418.93 last 23.62) - open/close 10254 (time 1788314864.05 total 428.94 last 25.37) - open/close 10502 (time 1788314874.06 total 438.95 last 24.78) - open/close 10751 (time 1788314884.06 total 448.95 last 24.89) - open/close 10960 (time 1788314896.15 total 461.04 last 17.29) - open/close 11248 (time 1788314906.86 total 471.75 last 26.89) - open/close 11529 (time 1788314917.50 total 482.39 last 26.42) - open/close 11808 (time 1788314927.51 total 492.40 last 27.85) - open/close 12057 (time 1788314937.54 total 502.43 last 24.85) - open/close 12327 (time 1788314948.27 total 513.16 last 25.14) - open/close 12596 (time 1788314958.28 total 523.17 last 26.87) - open/close 12784 (time 1788314969.70 total 534.59 last 16.47) - open/close 13062 (time 1788314980.27 total 545.16 last 26.30) - open/close 13345 (time 1788314990.28 total 555.17 last 28.27) - open/close 13605 (time 1788315001.60 total 566.49 last 22.96) - open/close 13868 (time 1788315011.95 total 576.84 last 25.42) - open/close 14126 (time 1788315022.18 total 587.07 last 25.21) - open/close 14403 (time 1788315032.18 total 597.07 last 27.70) - open/close 14660 (time 1788315042.19 total 607.08 last 25.68) - open/close 14843 (time 1788315052.20 total 617.09 last 18.28) - open/close 15083 (time 1788315062.21 total 627.10 last 23.98) - open/close 15345 (time 1788315074.67 total 639.56 last 21.03) - open/close 15586 (time 1788315084.97 total 649.86 last 23.39) - open/close 15830 (time 1788315094.99 total 659.88 last 24.35) - open/close 16009 (time 1788315105.61 total 670.50 last 16.86) - open/close 16260 (time 1788315116.35 total 681.24 last 23.36) - open/close 16447 (time 1788315126.44 total 691.33 last 18.53) - open/close 16695 (time 1788315136.55 total 701.44 last 24.53) - open/close 16955 (time 1788315146.85 total 711.74 last 25.25) - open/close 17205 (time 1788315156.86 total 721.75 last 24.97) - open/close 17491 (time 1788315167.33 total 732.22 last 27.33) - open/close 17744 (time 1788315178.17 total 743.06 last 23.32) - open/close 17983 (time 1788315188.18 total 753.07 last 23.89) - open/close 18211 (time 1788315198.95 total 763.84 last 21.17) - open/close 18498 (time 1788315208.95 total 773.84 last 28.69) - open/close 18635 (time 1788315219.93 total 784.82 last 12.47) - open/close 18905 (time 1788315230.47 total 795.36 last 25.62) - open/close 19120 (time 1788315240.54 total 805.43 last 21.36) - open/close 19279 (time 1788315250.82 total 815.71 last 15.46) - open/close 19553 (time 1788315261.80 total 826.69 last 24.95) - open/close 19829 (time 1788315272.36 total 837.25 last 26.12) - open/close 20000 (time 1788315278.02 total 842.91 last 30.25) - open/close 20768 (time 1788315288.03 total 852.92 last 76.70) - open/close 21519 (time 1788315298.04 total 862.93 last 75.03) - open/close 22305 (time 1788315308.04 total 872.93 last 78.59) - open/close 23089 (time 1788315318.05 total 882.94 last 78.34) - open/close 23827 (time 1788315328.06 total 892.95 last 73.73) - open/close 24637 (time 1788315338.06 total 902.95 last 80.95) - open/close 25306 (time 1788315348.07 total 912.96 last 66.86) - open/close 26113 (time 1788315358.08 total 922.97 last 80.64) - open/close 26910 (time 1788315368.08 total 932.97 last 79.66) - open/close 27705 (time 1788315378.08 total 942.97 last 79.50) - open/close 28545 (time 1788315388.08 total 952.97 last 84.00) - open/close 29281 (time 1788315398.09 total 962.98 last 73.57) - open/close 30000 (time 1788315406.99 total 971.88 last 80.77) - open/close 30840 (time 1788315416.99 total 981.88 last 83.99) - open/close 31650 (time 1788315427.00 total 991.89 last 80.94) - open/close 32495 (time 1788315437.01 total 1001.90 last 84.43) - open/close 33314 (time 1788315447.02 total 1011.91 last 81.83) - open/close 34037 (time 1788315457.02 total 1021.92 last 72.24) - open/close 34628 (time 1788315467.04 total 1031.93 last 59.02) - open/close 35279 (time 1788315477.05 total 1041.94 last 65.06) - open/close 36003 (time 1788315487.05 total 1051.94 last 72.36) - open/close 36717 (time 1788315497.06 total 1061.95 last 71.31) - open/close 37592 (time 1788315507.07 total 1071.96 last 87.48) - open/close 38413 (time 1788315517.08 total 1081.97 last 82.02) - open/close 39170 (time 1788315527.08 total 1091.97 last 75.69) - open/close 39815 (time 1788315537.10 total 1101.99 last 64.38) - open/close 40000 (time 1788315539.72 total 1104.61 last 70.45) - open/close 40733 (time 1788315549.73 total 1114.62 last 73.23) - open/close 41490 (time 1788315559.74 total 1124.63 last 75.65) - open/close 42280 (time 1788315569.75 total 1134.64 last 78.90) - open/close 42987 (time 1788315579.77 total 1144.66 last 70.58) - open/close 43790 (time 1788315589.77 total 1154.66 last 80.30) - open/close 44447 (time 1788315599.77 total 1164.66 last 65.68) - open/close 45232 (time 1788315609.77 total 1174.66 last 78.49) - open/close 46089 (time 1788315619.78 total 1184.67 last 85.69) - open/close 46856 (time 1788315629.78 total 1194.67 last 76.67) - open/close 47619 (time 1788315639.78 total 1204.67 last 76.29) - open/close 48423 (time 1788315649.78 total 1214.67 last 80.37) - open/close 49147 (time 1788315659.79 total 1224.68 last 72.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1230.95 seconds: 40.32 ops/second - unlinked 0 (time 1788315670 ; total 0 ; last 0) - unlinked 10000 (time 1788315801 ; total 131 ; last 131) - unlinked 20000 (time 1788315922 ; total 252 ; last 121) - unlinked 30000 (time 1788316048 ; total 378 ; last 126) - unlinked 40000 (time 1788316173 ; total 503 ; last 125) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 621 seconds: 79.921097 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg608-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg608-server unloading modules via unload_modules_local on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing unload_modules_local oleg608-server: modules unloaded. pdsh@oleg608-client: oleg608-server: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-client: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-client: ssh exited with exit code 2 pdsh@oleg608-client: oleg608-server: ssh exited with exit code 2