== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:19:42 (1788157182) start mds service on oleg625-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing load_modules_local oleg625-server: Loading modules from /home/green/git/lustre-release/lustre oleg625-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg625-server: detected 4 online CPUs by sysfs oleg625-server: MODOPTS_LIBCFS= oleg625-server: Force libcfs to create 2 CPU partitions oleg625-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg625-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg625-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg625-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg625-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1788157292.25 total 10.36 last 14.68) - open/close 345 (time 1788157302.25 total 20.36 last 19.29) - open/close 677 (time 1788157312.26 total 30.37 last 33.18) - open/close 812 (time 1788157323.10 total 41.21 last 12.45) - open/close 1124 (time 1788157333.73 total 51.84 last 29.36) - open/close 1337 (time 1788157343.74 total 61.85 last 21.28) - open/close 1566 (time 1788157354.72 total 72.83 last 20.85) - open/close 1709 (time 1788157364.73 total 82.84 last 14.29) - open/close 1962 (time 1788157375.51 total 93.62 last 23.47) - open/close 2174 (time 1788157386.29 total 104.40 last 19.67) - open/close 2395 (time 1788157396.30 total 114.41 last 22.08) - open/close 2682 (time 1788157406.90 total 125.00 last 27.09) - open/close 2933 (time 1788157416.91 total 135.02 last 25.06) - open/close 3146 (time 1788157426.91 total 145.02 last 21.29) - open/close 3395 (time 1788157436.92 total 155.02 last 24.89) - open/close 3685 (time 1788157446.93 total 165.03 last 28.97) - open/close 3939 (time 1788157457.17 total 175.28 last 24.79) - open/close 4177 (time 1788157467.99 total 186.10 last 21.99) - open/close 4323 (time 1788157478.43 total 196.54 last 13.99) - open/close 4435 (time 1788157488.45 total 206.56 last 11.17) - open/close 4654 (time 1788157499.44 total 217.55 last 19.93) - open/close 4806 (time 1788157511.69 total 229.80 last 12.40) - open/close 5018 (time 1788157522.04 total 240.15 last 20.49) - open/close 5102 (time 1788157532.13 total 250.24 last 8.32) - open/close 5296 (time 1788157543.66 total 261.76 last 16.84) - open/close 5462 (time 1788157554.15 total 272.25 last 15.82) - open/close 5778 (time 1788157565.15 total 283.26 last 28.73) - open/close 5952 (time 1788157575.44 total 293.55 last 16.91) - open/close 6155 (time 1788157585.49 total 303.60 last 20.19) - open/close 6299 (time 1788157595.73 total 313.84 last 14.07) - open/close 6523 (time 1788157605.73 total 323.84 last 22.39) - open/close 6739 (time 1788157615.77 total 333.88 last 21.52) - open/close 6956 (time 1788157626.83 total 344.94 last 19.62) - open/close 7221 (time 1788157637.13 total 355.24 last 25.71) - open/close 7504 (time 1788157647.92 total 366.03 last 26.23) - open/close 7659 (time 1788157658.21 total 376.32 last 15.07) - open/close 7934 (time 1788157668.23 total 386.34 last 27.44) - open/close 8103 (time 1788157678.29 total 396.39 last 16.80) - open/close 8305 (time 1788157688.60 total 406.71 last 19.58) - open/close 8630 (time 1788157698.83 total 416.94 last 31.78) - open/close 8869 (time 1788157708.88 total 426.99 last 23.78) - open/close 9074 (time 1788157719.52 total 437.63 last 19.26) - open/close 9364 (time 1788157729.87 total 447.98 last 28.02) - open/close 9616 (time 1788157740.52 total 458.62 last 23.68) - open/close 9763 (time 1788157750.52 total 468.63 last 14.69) - open/close 9949 (time 1788157763.60 total 481.71 last 14.22) - open/close 10000 (time 1788157766.57 total 484.68 last 17.14) - open/close 10122 (time 1788157777.11 total 495.22 last 11.58) - open/close 10303 (time 1788157787.23 total 505.34 last 17.89) - open/close 10549 (time 1788157797.24 total 515.35 last 24.57) - open/close 10853 (time 1788157807.69 total 525.80 last 29.10) - open/close 11011 (time 1788157817.70 total 535.81 last 15.79) - open/close 11224 (time 1788157828.13 total 546.23 last 20.43) - open/close 11480 (time 1788157838.14 total 556.25 last 25.56) - open/close 11767 (time 1788157848.87 total 566.98 last 26.74) - open/close 11992 (time 1788157858.89 total 577.00 last 22.46) - open/close 12253 (time 1788157869.07 total 587.18 last 25.65) - open/close 12521 (time 1788157879.40 total 597.50 last 25.95) - open/close 12737 (time 1788157890.01 total 608.12 last 20.35) - open/close 12887 (time 1788157902.15 total 620.26 last 12.35) - open/close 13076 (time 1788157913.11 total 631.21 last 17.26) - open/close 13300 (time 1788157923.13 total 641.24 last 22.34) - open/close 13395 (time 1788157933.14 total 651.25 last 9.49) - open/close 13549 (time 1788157943.84 total 661.95 last 14.40) - open/close 13788 (time 1788157954.32 total 672.43 last 22.81) - open/close 14057 (time 1788157964.32 total 682.43 last 26.89) - open/close 14305 (time 1788157975.18 total 693.29 last 22.84) - open/close 14578 (time 1788157985.19 total 703.29 last 27.28) - open/close 14732 (time 1788157995.44 total 713.55 last 15.02) - open/close 14975 (time 1788158006.37 total 724.48 last 22.23) - open/close 15259 (time 1788158016.90 total 735.00 last 26.99) - open/close 15563 (time 1788158027.42 total 745.53 last 28.90) - open/close 15809 (time 1788158037.42 total 755.53 last 24.58) - open/close 16071 (time 1788158047.77 total 765.88 last 25.31) - open/close 16375 (time 1788158058.36 total 776.46 last 28.73) - open/close 16683 (time 1788158069.09 total 787.20 last 28.69) - open/close 16955 (time 1788158079.95 total 798.06 last 25.05) - open/close 17254 (time 1788158089.95 total 808.06 last 29.88) - open/close 17437 (time 1788158100.17 total 818.28 last 17.91) - open/close 17615 (time 1788158110.17 total 828.28 last 17.80) - open/close 17808 (time 1788158120.19 total 838.30 last 19.28) - open/close 18037 (time 1788158130.20 total 848.30 last 22.88) - open/close 18227 (time 1788158141.15 total 859.26 last 17.34) - open/close 18349 (time 1788158151.18 total 869.29 last 12.17) - open/close 18510 (time 1788158161.31 total 879.42 last 15.88) - open/close 18713 (time 1788158171.35 total 889.46 last 20.22) - open/close 18947 (time 1788158182.40 total 900.51 last 21.19) - open/close 19137 (time 1788158201.17 total 919.28 last 10.12) - open/close 19437 (time 1788158211.46 total 929.56 last 29.18) - open/close 19684 (time 1788158221.84 total 939.95 last 23.78) - open/close 19997 (time 1788158234.59 total 952.70 last 24.55) - open/close 20237 (time 1788158244.59 total 962.70 last 23.99) - open/close 20487 (time 1788158254.60 total 972.71 last 24.98) - open/close 21336 (time 1788158264.61 total 982.72 last 84.80) - open/close 22333 (time 1788158274.61 total 992.72 last 99.68) - open/close 23271 (time 1788158284.62 total 1002.72 last 93.78) - open/close 24270 (time 1788158294.62 total 1012.73 last 99.82) - open/close 25391 (time 1788158304.64 total 1022.75 last 111.93) - open/close 26396 (time 1788158314.64 total 1032.75 last 100.45) - open/close 27327 (time 1788158324.65 total 1042.76 last 93.04) - open/close 28091 (time 1788158334.68 total 1052.79 last 76.15) - open/close 28758 (time 1788158344.69 total 1062.79 last 66.69) - open/close 29665 (time 1788158354.69 total 1072.80 last 90.68) - open/close 30000 (time 1788158358.16 total 1076.26 last 96.58) - open/close 30928 (time 1788158368.16 total 1086.27 last 92.79) - open/close 31889 (time 1788158378.17 total 1096.28 last 96.01) - open/close 32837 (time 1788158388.17 total 1106.28 last 94.76) - open/close 33789 (time 1788158398.17 total 1116.28 last 95.18) - open/close 34735 (time 1788158408.17 total 1126.28 last 94.59) - open/close 35702 (time 1788158418.18 total 1136.29 last 96.63) - open/close 36741 (time 1788158428.18 total 1146.29 last 103.89) - open/close 37698 (time 1788158438.18 total 1156.29 last 95.69) - open/close 38643 (time 1788158448.19 total 1166.30 last 94.47) - open/close 39609 (time 1788158458.20 total 1176.31 last 96.48) - open/close 40000 (time 1788158461.94 total 1180.05 last 104.57) - open/close 41029 (time 1788158471.95 total 1190.06 last 102.80) - open/close 41952 (time 1788158481.97 total 1200.08 last 92.13) - open/close 42832 (time 1788158491.97 total 1210.08 last 87.96) - open/close 43783 (time 1788158501.98 total 1220.09 last 95.02) - open/close 44702 (time 1788158511.99 total 1230.10 last 91.81) - open/close 45616 (time 1788158521.99 total 1240.10 last 91.39) - open/close 46558 (time 1788158531.99 total 1250.10 last 94.17) - open/close 47502 (time 1788158542.00 total 1260.11 last 94.33) - open/close 48507 (time 1788158552.01 total 1270.12 last 100.43) - open/close 49439 (time 1788158562.02 total 1280.12 last 93.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1282.10 seconds: 38.71 ops/second - unlinked 0 (time 1788158568 ; total 0 ; last 0) - unlinked 10000 (time 1788158689 ; total 121 ; last 121) - unlinked 20000 (time 1788158801 ; total 233 ; last 112) - unlinked 30000 (time 1788158863 ; total 295 ; last 62) - unlinked 40000 (time 1788158924 ; total 356 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 417 seconds: 119.019188 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg625-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg625-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg625-server unloading modules via unload_modules_local on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing unload_modules_local oleg625-server: modules unloaded. pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2