== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:23:48 (1787120628) start mds service on oleg123-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg123-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg123-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1787120766.79 total 10.01 last 24.28) - open/close 449 (time 1787120777.29 total 20.52 last 19.61) - open/close 694 (time 1787120787.30 total 30.52 last 24.48) - open/close 900 (time 1787120798.29 total 41.52 last 18.74) - open/close 1181 (time 1787120811.40 total 54.62 last 21.44) - open/close 1468 (time 1787120822.16 total 65.38 last 26.67) - open/close 1694 (time 1787120832.85 total 76.07 last 21.15) - open/close 1891 (time 1787120843.71 total 86.94 last 18.13) - open/close 2107 (time 1787120854.63 total 97.85 last 19.79) - open/close 2261 (time 1787120864.65 total 107.87 last 15.38) - open/close 2545 (time 1787120876.26 total 119.48 last 24.46) - open/close 2707 (time 1787120886.34 total 129.56 last 16.06) - open/close 2919 (time 1787120896.79 total 140.01 last 20.30) - open/close 3218 (time 1787120906.79 total 150.01 last 29.89) - open/close 3407 (time 1787120916.79 total 160.02 last 18.89) - open/close 3485 (time 1787120927.70 total 170.92 last 7.15) - open/close 3718 (time 1787120938.64 total 181.86 last 21.30) - open/close 3783 (time 1787120949.60 total 192.82 last 5.93) - open/close 3943 (time 1787120959.95 total 203.17 last 15.46) - open/close 4212 (time 1787120969.97 total 213.19 last 26.86) - open/close 4402 (time 1787120980.87 total 224.09 last 17.43) - open/close 4664 (time 1787120991.63 total 234.85 last 24.34) - open/close 4922 (time 1787121002.00 total 245.22 last 24.87) - open/close 5126 (time 1787121014.60 total 257.82 last 16.19) - open/close 5376 (time 1787121024.67 total 267.89 last 24.82) - open/close 5558 (time 1787121034.92 total 278.14 last 17.76) - open/close 5790 (time 1787121044.92 total 288.14 last 23.19) - open/close 5937 (time 1787121055.05 total 298.28 last 14.51) - open/close 6077 (time 1787121065.06 total 308.28 last 14.00) - open/close 6274 (time 1787121075.07 total 318.29 last 19.68) - open/close 6412 (time 1787121085.40 total 328.62 last 13.36) - open/close 6616 (time 1787121096.26 total 339.48 last 18.78) - open/close 6899 (time 1787121107.16 total 350.38 last 25.97) - open/close 7204 (time 1787121117.90 total 361.12 last 28.40) - open/close 7516 (time 1787121127.91 total 371.13 last 31.18) - open/close 7689 (time 1787121139.53 total 382.75 last 14.89) - open/close 7818 (time 1787121149.54 total 392.76 last 12.88) - open/close 7990 (time 1787121160.16 total 403.39 last 16.19) - open/close 8197 (time 1787121170.18 total 413.40 last 20.66) - open/close 8356 (time 1787121180.23 total 423.45 last 15.83) - open/close 8639 (time 1787121196.31 total 439.53 last 17.60) - open/close 8755 (time 1787121206.31 total 449.54 last 11.59) - open/close 8888 (time 1787121216.33 total 459.55 last 13.28) - open/close 9120 (time 1787121227.05 total 470.27 last 21.64) - open/close 9191 (time 1787121237.89 total 481.11 last 6.55) - open/close 9436 (time 1787121247.91 total 491.13 last 24.46) - open/close 9673 (time 1787121258.71 total 501.93 last 21.94) - open/close 9805 (time 1787121268.81 total 512.03 last 13.07) - open/close 9903 (time 1787121279.49 total 522.72 last 9.17) - open/close 10000 (time 1787121283.87 total 527.09 last 22.17) - open/close 10124 (time 1787121293.88 total 537.10 last 12.39) - open/close 10319 (time 1787121304.19 total 547.41 last 18.91) - open/close 10514 (time 1787121314.40 total 557.62 last 19.10) - open/close 10753 (time 1787121324.41 total 567.63 last 23.87) - open/close 10856 (time 1787121334.79 total 578.01 last 9.92) - open/close 11075 (time 1787121345.56 total 588.78 last 20.35) - open/close 11255 (time 1787121355.56 total 598.78 last 17.99) - open/close 11464 (time 1787121366.17 total 609.39 last 19.70) - open/close 11630 (time 1787121376.18 total 619.40 last 16.58) - open/close 11742 (time 1787121386.19 total 629.42 last 11.19) - open/close 11976 (time 1787121397.15 total 640.37 last 21.35) - open/close 12169 (time 1787121407.25 total 650.47 last 19.11) - open/close 12388 (time 1787121418.32 total 661.54 last 19.79) - open/close 12560 (time 1787121429.09 total 672.31 last 15.97) - open/close 12807 (time 1787121439.10 total 682.32 last 24.67) - open/close 13054 (time 1787121449.12 total 692.34 last 24.65) - open/close 13140 (time 1787121459.39 total 702.61 last 8.38) - open/close 13370 (time 1787121469.40 total 712.62 last 22.97) - open/close 13484 (time 1787121479.40 total 722.63 last 11.40) - open/close 13589 (time 1787121489.41 total 732.63 last 10.50) - open/close 13692 (time 1787121500.32 total 743.54 last 9.44) - open/close 13808 (time 1787121510.72 total 753.94 last 11.15) - open/close 13978 (time 1787121521.16 total 764.38 last 16.29) - open/close 14139 (time 1787121533.84 total 777.06 last 12.70) - open/close 14276 (time 1787121544.10 total 787.32 last 13.35) - open/close 14402 (time 1787121554.98 total 798.20 last 11.58) - open/close 14645 (time 1787121565.31 total 808.53 last 23.51) - open/close 14803 (time 1787121577.99 total 821.21 last 12.46) - open/close 14917 (time 1787121592.79 total 836.01 last 7.70) - open/close 15102 (time 1787121603.40 total 846.62 last 17.43) - open/close 15284 (time 1787121613.42 total 856.64 last 18.16) - open/close 15416 (time 1787121623.42 total 866.64 last 13.20) - open/close 15667 (time 1787121634.31 total 877.53 last 23.06) - open/close 15904 (time 1787121645.21 total 888.43 last 21.74) - open/close 16168 (time 1787121655.51 total 898.73 last 25.63) - open/close 16329 (time 1787121666.33 total 909.55 last 14.87) - open/close 16548 (time 1787121676.97 total 920.19 last 20.59) - open/close 16721 (time 1787121686.97 total 930.19 last 17.29) - open/close 16766 (time 1787121697.57 total 940.80 last 4.24) - open/close 16852 (time 1787121708.11 total 951.33 last 8.16) - open/close 17023 (time 1787121718.13 total 961.35 last 17.07) - open/close 17171 (time 1787121728.40 total 971.62 last 14.42) - open/close 17367 (time 1787121739.41 total 982.63 last 17.80) - open/close 17491 (time 1787121749.42 total 992.64 last 12.39) - open/close 17602 (time 1787121759.44 total 1002.66 last 11.08) - open/close 17680 (time 1787121769.75 total 1012.97 last 7.56) - open/close 17767 (time 1787121779.77 total 1022.99 last 8.69) - open/close 17948 (time 1787121790.42 total 1033.64 last 16.99) - open/close 18128 (time 1787121800.59 total 1043.81 last 17.71) - open/close 18360 (time 1787121810.60 total 1053.82 last 23.16) - open/close 19066 (time 1787121820.60 total 1063.82 last 70.60) - open/close 19899 (time 1787121830.61 total 1073.83 last 83.23) - open/close 20707 (time 1787121840.61 total 1083.83 last 80.79) - open/close 21465 (time 1787121850.62 total 1093.84 last 75.74) - open/close 22198 (time 1787121860.65 total 1103.87 last 73.09) - open/close 23003 (time 1787121870.65 total 1113.88 last 80.46) - open/close 23688 (time 1787121880.66 total 1123.88 last 68.48) - open/close 24469 (time 1787121890.67 total 1133.89 last 77.99) - open/close 25417 (time 1787121900.67 total 1143.89 last 94.78) - open/close 26150 (time 1787121910.68 total 1153.91 last 73.21) - open/close 26944 (time 1787121920.70 total 1163.93 last 79.25) - open/close 27705 (time 1787121930.71 total 1173.93 last 76.06) - open/close 28518 (time 1787121940.71 total 1183.93 last 81.29) - open/close 29393 (time 1787121950.71 total 1193.93 last 87.49) - open/close 30000 (time 1787121957.64 total 1200.86 last 87.66) - open/close 30790 (time 1787121967.64 total 1210.86 last 79.00) - open/close 31398 (time 1787121977.65 total 1220.87 last 60.74) - open/close 32012 (time 1787121987.65 total 1230.87 last 61.38) - open/close 32752 (time 1787121997.65 total 1240.87 last 73.99) - open/close 33588 (time 1787122007.66 total 1250.88 last 83.54) - open/close 34287 (time 1787122017.67 total 1260.89 last 69.84) - open/close 35092 (time 1787122027.68 total 1270.90 last 80.42) - open/close 35881 (time 1787122037.69 total 1280.92 last 78.76) - open/close 36541 (time 1787122047.70 total 1290.93 last 65.94) - open/close 37043 (time 1787122057.71 total 1300.93 last 50.16) - open/close 37846 (time 1787122067.75 total 1310.97 last 80.02) - open/close 38522 (time 1787122077.76 total 1320.98 last 67.52) - open/close 39249 (time 1787122087.77 total 1330.99 last 72.64) - open/close 39879 (time 1787122097.80 total 1341.02 last 62.81) - open/close 40000 (time 1787122100.15 total 1343.37 last 51.50) - open/close 40669 (time 1787122110.15 total 1353.37 last 66.87) - open/close 41369 (time 1787122120.17 total 1363.39 last 69.88) - open/close 42130 (time 1787122130.18 total 1373.40 last 76.03) - open/close 42910 (time 1787122140.18 total 1383.41 last 77.95) - open/close 43788 (time 1787122150.19 total 1393.41 last 87.76) - open/close 44525 (time 1787122160.20 total 1403.42 last 73.64) - open/close 45233 (time 1787122170.20 total 1413.42 last 70.77) - open/close 45804 (time 1787122180.20 total 1423.43 last 57.08) - open/close 46468 (time 1787122190.22 total 1433.44 last 66.29) - open/close 47098 (time 1787122200.23 total 1443.45 last 62.97) - open/close 47791 (time 1787122210.23 total 1453.45 last 69.26) - open/close 48501 (time 1787122220.25 total 1463.47 last 70.89) - open/close 49106 (time 1787122230.26 total 1473.48 last 60.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1483.24 seconds: 33.46 ops/second - unlinked 0 (time 1787122245 ; total 0 ; last 0) - unlinked 10000 (time 1787122383 ; total 138 ; last 138) - unlinked 20000 (time 1787122536 ; total 291 ; last 153) - unlinked 30000 (time 1787122682 ; total 437 ; last 146) - unlinked 40000 (time 1787122818 ; total 573 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 707 seconds: 70.199432 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2