== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:58:57 (1780340337) start mds service on oleg223-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg223-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 211 (time 1780340463.48 total 11.03 last 19.13) - open/close 493 (time 1780340473.76 total 21.30 last 27.45) - open/close 689 (time 1780340484.61 total 32.16 last 18.06) - open/close 833 (time 1780340495.31 total 42.86 last 13.45) - open/close 1056 (time 1780340508.31 total 55.86 last 17.15) - open/close 1279 (time 1780340521.11 total 68.66 last 17.42) - open/close 1406 (time 1780340532.11 total 79.66 last 11.54) - open/close 1579 (time 1780340544.17 total 91.72 last 14.35) - open/close 1685 (time 1780340554.52 total 102.07 last 10.24) - open/close 1818 (time 1780340564.54 total 112.09 last 13.27) - open/close 1966 (time 1780340574.55 total 122.10 last 14.79) - open/close 2176 (time 1780340586.13 total 133.68 last 18.12) - open/close 2360 (time 1780340596.82 total 144.37 last 17.21) - open/close 2539 (time 1780340607.89 total 155.44 last 16.17) - open/close 2784 (time 1780340618.50 total 166.05 last 23.08) - open/close 2956 (time 1780340629.14 total 176.69 last 16.18) - open/close 3011 (time 1780340639.76 total 187.31 last 5.18) - open/close 3124 (time 1780340650.66 total 198.21 last 10.37) - open/close 3201 (time 1780340660.89 total 208.44 last 7.52) - open/close 3321 (time 1780340670.92 total 218.47 last 11.96) - open/close 3464 (time 1780340681.86 total 229.41 last 13.08) - open/close 3629 (time 1780340691.87 total 239.42 last 16.49) - open/close 3681 (time 1780340702.53 total 250.08 last 4.88) - open/close 3796 (time 1780340712.54 total 260.09 last 11.49) - open/close 3891 (time 1780340722.91 total 270.46 last 9.16) - open/close 4063 (time 1780340735.26 total 282.81 last 13.93) - open/close 4296 (time 1780340745.85 total 293.40 last 21.99) - open/close 4470 (time 1780340756.83 total 304.38 last 15.85) - open/close 4605 (time 1780340767.69 total 315.24 last 12.43) - open/close 4739 (time 1780340778.66 total 326.21 last 12.22) - open/close 4846 (time 1780340788.68 total 336.23 last 10.68) - open/close 5060 (time 1780340799.43 total 346.98 last 19.90) - open/close 5317 (time 1780340810.00 total 357.55 last 24.31) - open/close 5430 (time 1780340820.56 total 368.11 last 10.70) - open/close 5600 (time 1780340831.25 total 378.80 last 15.91) - open/close 5838 (time 1780340841.94 total 389.49 last 22.26) - open/close 6000 (time 1780340852.12 total 399.67 last 15.92) - open/close 6203 (time 1780340862.12 total 409.67 last 20.29) - open/close 6299 (time 1780340872.84 total 420.39 last 8.96) - open/close 6441 (time 1780340882.84 total 430.39 last 14.20) - open/close 6543 (time 1780340892.90 total 440.44 last 10.15) - open/close 6639 (time 1780340903.73 total 451.28 last 8.86) - open/close 6853 (time 1780340914.13 total 461.68 last 20.59) - open/close 7031 (time 1780340924.13 total 471.68 last 17.79) - open/close 7277 (time 1780340934.13 total 481.68 last 24.59) - open/close 7498 (time 1780340944.43 total 491.98 last 21.47) - open/close 7691 (time 1780340955.21 total 502.76 last 17.91) - open/close 7849 (time 1780340965.57 total 513.12 last 15.24) - open/close 7968 (time 1780340976.54 total 524.09 last 10.85) - open/close 8063 (time 1780340986.86 total 534.41 last 9.21) - open/close 8212 (time 1780340997.13 total 544.68 last 14.51) - open/close 8396 (time 1780341007.14 total 554.69 last 18.38) - open/close 8570 (time 1780341017.93 total 565.48 last 16.13) - open/close 8698 (time 1780341028.33 total 575.88 last 12.31) - open/close 8913 (time 1780341038.67 total 586.22 last 20.79) - open/close 9103 (time 1780341048.69 total 596.24 last 18.95) - open/close 9184 (time 1780341059.22 total 606.77 last 7.70) - open/close 9256 (time 1780341070.27 total 617.82 last 6.52) - open/close 9388 (time 1780341080.27 total 627.82 last 13.19) - open/close 9545 (time 1780341090.29 total 637.84 last 15.68) - open/close 9674 (time 1780341100.77 total 648.32 last 12.31) - open/close 9895 (time 1780341111.17 total 658.72 last 21.24) - open/close 10000 (time 1780341118.72 total 666.27 last 13.91) - open/close 10048 (time 1780341129.50 total 677.05 last 4.45) - open/close 10141 (time 1780341140.16 total 687.71 last 8.72) - open/close 10215 (time 1780341150.17 total 697.72 last 7.40) - open/close 10335 (time 1780341160.19 total 707.73 last 11.98) - open/close 10409 (time 1780341170.34 total 717.89 last 7.29) - open/close 10589 (time 1780341180.43 total 727.98 last 17.84) - open/close 10732 (time 1780341190.45 total 738.00 last 14.27) - open/close 10966 (time 1780341201.20 total 748.75 last 21.76) - open/close 11079 (time 1780341211.60 total 759.15 last 10.86) - open/close 11233 (time 1780341222.03 total 769.58 last 14.77) - open/close 11409 (time 1780341232.58 total 780.13 last 16.68) - open/close 11652 (time 1780341242.69 total 790.24 last 24.03) - open/close 11917 (time 1780341252.71 total 800.26 last 26.46) - open/close 12141 (time 1780341263.68 total 811.23 last 20.42) - open/close 12365 (time 1780341274.64 total 822.19 last 20.43) - open/close 12656 (time 1780341284.93 total 832.48 last 28.28) - open/close 12926 (time 1780341295.50 total 843.05 last 25.56) - open/close 13212 (time 1780341306.00 total 853.55 last 27.23) - open/close 13421 (time 1780341316.00 total 863.55 last 20.89) - open/close 13664 (time 1780341326.41 total 873.96 last 23.35) - open/close 13875 (time 1780341336.42 total 883.97 last 21.08) - open/close 13994 (time 1780341346.75 total 894.30 last 11.52) - open/close 14264 (time 1780341356.80 total 904.35 last 26.86) - open/close 14512 (time 1780341366.81 total 914.36 last 24.77) - open/close 14747 (time 1780341377.35 total 924.90 last 22.31) - open/close 15005 (time 1780341387.36 total 934.91 last 25.78) - open/close 15276 (time 1780341397.62 total 945.17 last 26.41) - open/close 15551 (time 1780341407.62 total 955.17 last 27.49) - open/close 15793 (time 1780341417.92 total 965.47 last 23.50) - open/close 16000 (time 1780341428.00 total 975.55 last 20.53) - open/close 16256 (time 1780341438.00 total 985.55 last 25.60) - open/close 16500 (time 1780341448.27 total 995.82 last 23.76) - open/close 16772 (time 1780341458.90 total 1006.45 last 25.59) - open/close 17031 (time 1780341471.57 total 1019.12 last 20.44) - open/close 17295 (time 1780341481.87 total 1029.42 last 25.63) - open/close 17557 (time 1780341491.97 total 1039.52 last 25.94) - open/close 17811 (time 1780341501.98 total 1049.53 last 25.39) - open/close 18277 (time 1780341511.99 total 1059.54 last 46.57) - open/close 19301 (time 1780341521.99 total 1069.54 last 102.35) - open/close 20000 (time 1780341529.51 total 1077.06 last 93.00) - open/close 20998 (time 1780341539.51 total 1087.06 last 99.75) - open/close 21932 (time 1780341549.51 total 1097.06 last 93.39) - open/close 22907 (time 1780341559.52 total 1107.07 last 97.40) - open/close 23814 (time 1780341569.53 total 1117.08 last 90.68) - open/close 24751 (time 1780341579.53 total 1127.08 last 93.64) - open/close 25764 (time 1780341589.53 total 1137.08 last 101.29) - open/close 26701 (time 1780341599.54 total 1147.09 last 93.66) - open/close 27652 (time 1780341609.55 total 1157.10 last 94.98) - open/close 28592 (time 1780341619.56 total 1167.11 last 93.93) - open/close 29687 (time 1780341629.56 total 1177.11 last 109.42) - open/close 30000 (time 1780341633.06 total 1180.61 last 89.45) - open/close 30862 (time 1780341643.08 total 1190.63 last 86.07) - open/close 31735 (time 1780341653.08 total 1200.63 last 87.29) - open/close 32626 (time 1780341663.12 total 1210.66 last 88.78) - open/close 33503 (time 1780341673.14 total 1220.69 last 87.45) - open/close 34259 (time 1780341683.17 total 1230.72 last 75.37) - open/close 35128 (time 1780341693.18 total 1240.73 last 86.88) - open/close 36048 (time 1780341703.18 total 1250.73 last 91.92) - open/close 37021 (time 1780341713.19 total 1260.74 last 97.29) - open/close 38014 (time 1780341723.19 total 1270.74 last 99.24) - open/close 39043 (time 1780341733.20 total 1280.75 last 102.78) - open/close 39984 (time 1780341743.20 total 1290.75 last 94.10) - open/close 40961 (time 1780341753.21 total 1300.76 last 97.67) - open/close 41990 (time 1780341763.21 total 1310.76 last 102.87) - open/close 43083 (time 1780341773.22 total 1320.77 last 109.20) - open/close 44122 (time 1780341783.24 total 1330.79 last 103.67) - open/close 45171 (time 1780341793.24 total 1340.79 last 104.85) - open/close 46116 (time 1780341803.25 total 1350.80 last 94.47) - open/close 47266 (time 1780341813.25 total 1360.80 last 114.99) - open/close 48412 (time 1780341823.26 total 1370.81 last 114.52) - open/close 49532 (time 1780341833.26 total 1380.81 last 111.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1381.74 seconds: 35.92 ops/second - unlinked 0 (time 1780341837 ; total 0 ; last 0) - unlinked 10000 (time 1780341931 ; total 94 ; last 94) - unlinked 20000 (time 1780342000 ; total 163 ; last 69) - unlinked 30000 (time 1780342056 ; total 219 ; last 56) - unlinked 40000 (time 1780342105 ; total 268 ; last 49) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 316 seconds: 157.060120 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded. pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2