== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:36:29 (1780338989) start mds service on oleg231-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg231-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg231-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg231-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1780339101.45 total 10.72 last 18.94) - open/close 328 (time 1780339112.43 total 21.69 last 11.39) - open/close 623 (time 1780339122.43 total 31.70 last 29.50) - open/close 856 (time 1780339133.31 total 42.58 last 21.41) - open/close 1079 (time 1780339143.31 total 52.58 last 22.30) - open/close 1295 (time 1780339153.34 total 62.61 last 21.53) - open/close 1499 (time 1780339163.35 total 72.62 last 20.38) - open/close 1627 (time 1780339176.27 total 85.54 last 9.91) - open/close 1727 (time 1780339186.88 total 96.15 last 9.43) - open/close 1837 (time 1780339196.95 total 106.22 last 10.93) - open/close 1906 (time 1780339206.95 total 116.22 last 6.90) - open/close 2094 (time 1780339216.96 total 126.23 last 18.79) - open/close 2323 (time 1780339227.22 total 136.49 last 22.31) - open/close 2449 (time 1780339237.96 total 147.23 last 11.73) - open/close 2501 (time 1780339250.67 total 159.94 last 4.09) - open/close 2585 (time 1780339261.26 total 170.53 last 7.93) - open/close 2660 (time 1780339271.27 total 180.53 last 7.50) - open/close 2837 (time 1780339281.27 total 190.53 last 17.70) - open/close 2963 (time 1780339291.39 total 200.66 last 12.44) - open/close 3073 (time 1780339302.29 total 211.56 last 10.10) - open/close 3244 (time 1780339312.29 total 221.56 last 17.09) - open/close 3333 (time 1780339322.63 total 231.90 last 8.61) - open/close 3425 (time 1780339332.65 total 241.92 last 9.18) - open/close 3721 (time 1780339343.61 total 252.88 last 27.00) - open/close 3917 (time 1780339353.62 total 262.89 last 19.58) - open/close 4089 (time 1780339364.99 total 274.25 last 15.14) - open/close 4326 (time 1780339374.99 total 284.26 last 23.68) - open/close 4489 (time 1780339386.01 total 295.28 last 14.80) - open/close 4671 (time 1780339396.02 total 305.29 last 18.18) - open/close 4857 (time 1780339406.05 total 315.32 last 18.55) - open/close 4973 (time 1780339416.05 total 325.32 last 11.60) - open/close 5089 (time 1780339426.06 total 335.33 last 11.59) - open/close 5277 (time 1780339436.76 total 346.03 last 17.57) - open/close 5406 (time 1780339446.77 total 356.03 last 12.89) - open/close 5545 (time 1780339457.29 total 366.56 last 13.21) - open/close 5614 (time 1780339469.32 total 378.58 last 5.74) - open/close 5888 (time 1780339479.97 total 389.24 last 25.72) - open/close 6059 (time 1780339490.89 total 400.16 last 15.66) - open/close 6139 (time 1780339500.91 total 410.17 last 7.99) - open/close 6273 (time 1780339510.93 total 420.19 last 13.37) - open/close 6384 (time 1780339520.95 total 430.22 last 11.07) - open/close 6494 (time 1780339531.33 total 440.60 last 10.60) - open/close 6677 (time 1780339541.33 total 450.60 last 18.29) - open/close 6741 (time 1780339551.35 total 460.61 last 6.39) - open/close 6850 (time 1780339561.36 total 470.63 last 10.88) - open/close 6926 (time 1780339571.99 total 481.26 last 7.15) - open/close 7055 (time 1780339582.02 total 491.29 last 12.87) - open/close 7224 (time 1780339593.10 total 502.37 last 15.25) - open/close 7425 (time 1780339603.34 total 512.61 last 19.64) - open/close 7642 (time 1780339613.35 total 522.62 last 21.67) - open/close 7943 (time 1780339623.36 total 532.63 last 30.07) - open/close 8208 (time 1780339633.38 total 542.64 last 26.46) - open/close 8473 (time 1780339644.35 total 553.61 last 24.16) - open/close 8768 (time 1780339654.92 total 564.19 last 27.88) - open/close 9012 (time 1780339665.81 total 575.08 last 22.41) - open/close 9159 (time 1780339676.51 total 585.78 last 13.75) - open/close 9264 (time 1780339687.25 total 596.51 last 9.78) - open/close 9420 (time 1780339698.05 total 607.32 last 14.43) - open/close 9615 (time 1780339708.64 total 617.91 last 18.42) - open/close 9792 (time 1780339718.64 total 627.91 last 17.69) - open/close 9922 (time 1780339728.80 total 638.06 last 12.81) - open/close 10000 (time 1780339734.25 total 643.52 last 14.29) - open/close 10210 (time 1780339744.44 total 653.71 last 20.61) - open/close 10450 (time 1780339754.52 total 663.79 last 23.82) - open/close 10647 (time 1780339765.46 total 674.73 last 18.01) - open/close 10809 (time 1780339775.95 total 685.22 last 15.44) - open/close 10978 (time 1780339785.98 total 695.25 last 16.85) - open/close 11166 (time 1780339795.99 total 705.26 last 18.77) - open/close 11365 (time 1780339806.01 total 715.28 last 19.86) - open/close 11488 (time 1780339817.39 total 726.66 last 10.81) - open/close 11639 (time 1780339828.42 total 737.69 last 13.69) - open/close 11786 (time 1780339838.72 total 747.99 last 14.27) - open/close 12009 (time 1780339848.73 total 758.00 last 22.28) - open/close 12175 (time 1780339858.73 total 768.00 last 16.60) - open/close 12331 (time 1780339868.73 total 778.00 last 15.60) - open/close 12445 (time 1780339881.73 total 790.99 last 8.78) - open/close 12612 (time 1780339891.73 total 801.00 last 16.69) - open/close 12734 (time 1780339901.75 total 811.02 last 12.18) - open/close 12885 (time 1780339911.96 total 821.23 last 14.78) - open/close 13057 (time 1780339921.99 total 831.26 last 17.15) - open/close 13179 (time 1780339932.74 total 842.01 last 11.34) - open/close 13262 (time 1780339943.42 total 852.69 last 7.77) - open/close 13371 (time 1780339953.63 total 862.89 last 10.68) - open/close 13615 (time 1780339964.42 total 873.68 last 22.61) - open/close 13898 (time 1780339974.74 total 884.00 last 27.42) - open/close 14059 (time 1780339987.21 total 896.48 last 12.91) - open/close 14191 (time 1780339997.73 total 907.00 last 12.55) - open/close 14385 (time 1780340009.17 total 918.44 last 16.96) - open/close 14581 (time 1780340019.18 total 928.45 last 19.59) - open/close 14721 (time 1780340029.20 total 938.47 last 13.97) - open/close 14971 (time 1780340040.20 total 949.47 last 22.74) - open/close 15190 (time 1780340050.21 total 959.47 last 21.88) - open/close 15330 (time 1780340060.59 total 969.86 last 13.49) - open/close 15413 (time 1780340070.98 total 980.25 last 7.98) - open/close 15536 (time 1780340081.00 total 990.27 last 12.28) - open/close 15687 (time 1780340091.95 total 1001.22 last 13.79) - open/close 15834 (time 1780340103.01 total 1012.28 last 13.29) - open/close 16007 (time 1780340113.83 total 1023.10 last 15.99) - open/close 16177 (time 1780340123.97 total 1033.24 last 16.76) - open/close 16361 (time 1780340134.17 total 1043.44 last 18.04) - open/close 16526 (time 1780340144.17 total 1053.44 last 16.50) - open/close 16666 (time 1780340156.51 total 1065.77 last 11.35) - open/close 16767 (time 1780340166.92 total 1076.19 last 9.70) - open/close 17345 (time 1780340176.93 total 1086.20 last 57.73) - open/close 18073 (time 1780340186.94 total 1096.21 last 72.74) - open/close 18738 (time 1780340196.96 total 1106.22 last 66.39) - open/close 19441 (time 1780340206.96 total 1116.23 last 70.28) - open/close 20000 (time 1780340215.26 total 1124.53 last 67.34) - open/close 20479 (time 1780340225.32 total 1134.59 last 47.61) - open/close 20899 (time 1780340235.33 total 1144.60 last 41.95) - open/close 21326 (time 1780340245.36 total 1154.63 last 42.58) - open/close 21823 (time 1780340255.37 total 1164.64 last 49.66) - open/close 22265 (time 1780340265.38 total 1174.65 last 44.16) - open/close 22873 (time 1780340275.38 total 1184.65 last 60.79) - open/close 23410 (time 1780340285.39 total 1194.66 last 53.68) - open/close 23949 (time 1780340295.39 total 1204.66 last 53.87) - open/close 24734 (time 1780340305.40 total 1214.67 last 78.44) - open/close 25544 (time 1780340315.41 total 1224.68 last 80.89) - open/close 26122 (time 1780340325.43 total 1234.70 last 57.72) - open/close 26875 (time 1780340335.44 total 1244.70 last 75.24) - open/close 27478 (time 1780340345.44 total 1254.71 last 60.26) - open/close 28301 (time 1780340355.47 total 1264.73 last 82.11) - open/close 29088 (time 1780340365.47 total 1274.74 last 78.64) - open/close 29905 (time 1780340375.49 total 1284.76 last 81.52) - open/close 30674 (time 1780340385.51 total 1294.78 last 76.75) - open/close 31448 (time 1780340395.53 total 1304.79 last 77.32) - open/close 32252 (time 1780340405.53 total 1314.80 last 80.36) - open/close 33177 (time 1780340415.53 total 1324.80 last 92.49) - open/close 33937 (time 1780340425.53 total 1334.80 last 75.99) - open/close 34737 (time 1780340435.54 total 1344.81 last 79.94) - open/close 35368 (time 1780340445.54 total 1354.81 last 63.08) - open/close 36081 (time 1780340455.55 total 1364.82 last 71.25) - open/close 36875 (time 1780340465.56 total 1374.82 last 79.35) - open/close 37640 (time 1780340475.56 total 1384.83 last 76.46) - open/close 38280 (time 1780340485.57 total 1394.83 last 63.97) - open/close 38843 (time 1780340495.59 total 1404.86 last 56.18) - open/close 39392 (time 1780340505.59 total 1414.86 last 54.88) - open/close 40000 (time 1780340513.14 total 1422.41 last 80.56) - open/close 40966 (time 1780340523.14 total 1432.41 last 96.56) - open/close 41840 (time 1780340533.15 total 1442.41 last 87.38) - open/close 42546 (time 1780340543.16 total 1452.43 last 70.51) - open/close 43130 (time 1780340553.16 total 1462.43 last 58.39) - open/close 43677 (time 1780340563.17 total 1472.43 last 54.67) - open/close 44551 (time 1780340573.17 total 1482.44 last 87.34) - open/close 45403 (time 1780340583.17 total 1492.44 last 85.18) - open/close 46131 (time 1780340593.18 total 1502.45 last 72.74) - open/close 46744 (time 1780340603.19 total 1512.46 last 61.27) - open/close 47428 (time 1780340613.20 total 1522.47 last 68.29) - open/close 48056 (time 1780340623.22 total 1532.49 last 62.71) - open/close 48619 (time 1780340633.23 total 1542.50 last 56.25) - open/close 49300 (time 1780340643.24 total 1552.50 last 68.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1557.34 seconds: 31.87 ops/second - unlinked 0 (time 1780340654 ; total 0 ; last 0) - unlinked 10000 (time 1780340801 ; total 147 ; last 147) - unlinked 20000 (time 1780340953 ; total 299 ; last 152) - unlinked 30000 (time 1780341113 ; total 459 ; last 160) - unlinked 40000 (time 1780341255 ; total 601 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 715 seconds: 69.413986 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg231-server unloading modules via unload_modules_local on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2