== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:17:07 (1783952227) start mds service on oleg137-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg137-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg137-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1783952338.08 total 10.14 last 21.29) - open/close 473 (time 1783952348.51 total 20.57 last 24.65) - open/close 682 (time 1783952358.52 total 30.58 last 20.88) - open/close 912 (time 1783952368.94 total 41.00 last 22.07) - open/close 1118 (time 1783952378.96 total 51.02 last 20.55) - open/close 1357 (time 1783952388.98 total 61.04 last 23.87) - open/close 1613 (time 1783952398.99 total 71.05 last 25.57) - open/close 1793 (time 1783952409.59 total 81.65 last 16.99) - open/close 1996 (time 1783952419.59 total 91.65 last 20.29) - open/close 2258 (time 1783952429.61 total 101.67 last 26.16) - open/close 2402 (time 1783952439.84 total 111.90 last 14.07) - open/close 2620 (time 1783952450.35 total 122.41 last 20.74) - open/close 2697 (time 1783952460.36 total 132.42 last 7.70) - open/close 2814 (time 1783952470.37 total 142.43 last 11.68) - open/close 3011 (time 1783952482.69 total 154.75 last 15.99) - open/close 3140 (time 1783952493.16 total 165.22 last 12.33) - open/close 3269 (time 1783952503.73 total 175.79 last 12.20) - open/close 3477 (time 1783952513.90 total 185.96 last 20.46) - open/close 3714 (time 1783952526.24 total 198.30 last 19.20) - open/close 3979 (time 1783952536.61 total 208.67 last 25.56) - open/close 4164 (time 1783952546.62 total 218.68 last 18.49) - open/close 4440 (time 1783952556.62 total 228.68 last 27.59) - open/close 4601 (time 1783952566.63 total 238.69 last 16.09) - open/close 4867 (time 1783952576.65 total 248.71 last 26.55) - open/close 5029 (time 1783952587.01 total 259.07 last 15.64) - open/close 5310 (time 1783952597.02 total 269.08 last 28.05) - open/close 5501 (time 1783952607.04 total 279.10 last 19.07) - open/close 5681 (time 1783952617.05 total 289.11 last 17.99) - open/close 5798 (time 1783952628.60 total 300.66 last 10.13) - open/close 6105 (time 1783952638.60 total 310.66 last 30.69) - open/close 6297 (time 1783952648.60 total 320.66 last 19.20) - open/close 6575 (time 1783952659.05 total 331.11 last 26.61) - open/close 6671 (time 1783952669.28 total 341.34 last 9.39) - open/close 6915 (time 1783952679.79 total 351.85 last 23.22) - open/close 7097 (time 1783952690.75 total 362.81 last 16.60) - open/close 7217 (time 1783952700.78 total 372.84 last 11.96) - open/close 7342 (time 1783952711.81 total 383.87 last 11.33) - open/close 7533 (time 1783952721.82 total 393.88 last 19.10) - open/close 7788 (time 1783952732.74 total 404.80 last 23.35) - open/close 7950 (time 1783952742.74 total 414.80 last 16.19) - open/close 8163 (time 1783952752.75 total 424.81 last 21.28) - open/close 8314 (time 1783952763.17 total 435.23 last 14.49) - open/close 8514 (time 1783952773.19 total 445.25 last 19.96) - open/close 8751 (time 1783952783.20 total 455.26 last 23.68) - open/close 8933 (time 1783952793.95 total 466.01 last 16.92) - open/close 9056 (time 1783952805.68 total 477.74 last 10.49) - open/close 9265 (time 1783952816.18 total 488.24 last 19.90) - open/close 9445 (time 1783952826.20 total 498.26 last 17.98) - open/close 9708 (time 1783952836.41 total 508.47 last 25.74) - open/close 9878 (time 1783952846.88 total 518.94 last 16.24) - open/close 10000 (time 1783952853.18 total 525.24 last 19.37) - open/close 10184 (time 1783952863.53 total 535.59 last 17.79) - open/close 10393 (time 1783952873.53 total 545.59 last 20.89) - open/close 10617 (time 1783952883.74 total 555.80 last 21.94) - open/close 10880 (time 1783952894.62 total 566.68 last 24.18) - open/close 11086 (time 1783952905.46 total 577.52 last 19.00) - open/close 11323 (time 1783952915.52 total 587.58 last 23.55) - open/close 11461 (time 1783952926.18 total 598.24 last 12.96) - open/close 11713 (time 1783952936.19 total 608.25 last 25.16) - open/close 11851 (time 1783952946.73 total 618.79 last 13.10) - open/close 12024 (time 1783952956.74 total 628.80 last 17.28) - open/close 12100 (time 1783952968.56 total 640.62 last 6.43) - open/close 12328 (time 1783952978.56 total 650.62 last 22.79) - open/close 12496 (time 1783952993.47 total 665.53 last 11.27) - open/close 12579 (time 1783953004.27 total 676.33 last 7.68) - open/close 12732 (time 1783953015.09 total 687.15 last 14.14) - open/close 12978 (time 1783953025.82 total 697.88 last 22.93) - open/close 13188 (time 1783953036.16 total 708.22 last 20.31) - open/close 13377 (time 1783953049.15 total 721.21 last 14.55) - open/close 13470 (time 1783953059.64 total 731.70 last 8.86) - open/close 13543 (time 1783953069.74 total 741.80 last 7.23) - open/close 13740 (time 1783953079.81 total 751.87 last 19.58) - open/close 14028 (time 1783953089.81 total 761.87 last 28.78) - open/close 14265 (time 1783953099.82 total 771.88 last 23.69) - open/close 14518 (time 1783953110.58 total 782.64 last 23.50) - open/close 14727 (time 1783953120.59 total 792.65 last 20.89) - open/close 14853 (time 1783953132.26 total 804.32 last 10.80) - open/close 15075 (time 1783953142.50 total 814.56 last 21.67) - open/close 15281 (time 1783953152.53 total 824.59 last 20.55) - open/close 15496 (time 1783953162.54 total 834.60 last 21.47) - open/close 15740 (time 1783953172.55 total 844.61 last 24.38) - open/close 15884 (time 1783953182.55 total 854.62 last 14.40) - open/close 16149 (time 1783953192.56 total 864.62 last 26.48) - open/close 16433 (time 1783953202.58 total 874.64 last 28.35) - open/close 16669 (time 1783953213.28 total 885.34 last 22.05) - open/close 16783 (time 1783953223.98 total 896.04 last 10.66) - open/close 16899 (time 1783953234.47 total 906.53 last 11.05) - open/close 17005 (time 1783953245.45 total 917.51 last 9.66) - open/close 17126 (time 1783953255.76 total 927.82 last 11.73) - open/close 17287 (time 1783953266.81 total 938.87 last 14.57) - open/close 17463 (time 1783953277.70 total 949.76 last 16.15) - open/close 17590 (time 1783953287.72 total 959.78 last 12.68) - open/close 17662 (time 1783953297.72 total 969.78 last 7.20) - open/close 17857 (time 1783953308.16 total 980.22 last 18.69) - open/close 18005 (time 1783953318.92 total 990.98 last 13.75) - open/close 18184 (time 1783953328.92 total 1000.98 last 17.90) - open/close 18340 (time 1783953338.93 total 1010.99 last 15.59) - open/close 18979 (time 1783953348.94 total 1021.00 last 63.86) - open/close 19707 (time 1783953358.94 total 1031.00 last 72.75) - open/close 20000 (time 1783953363.02 total 1035.08 last 71.89) - open/close 20676 (time 1783953373.03 total 1045.09 last 67.51) - open/close 21354 (time 1783953383.04 total 1055.10 last 67.78) - open/close 21962 (time 1783953393.04 total 1065.10 last 60.78) - open/close 22579 (time 1783953403.05 total 1075.11 last 61.64) - open/close 23211 (time 1783953413.06 total 1085.12 last 63.15) - open/close 23966 (time 1783953423.08 total 1095.14 last 75.35) - open/close 24646 (time 1783953433.09 total 1105.15 last 67.93) - open/close 25294 (time 1783953443.10 total 1115.16 last 64.73) - open/close 25977 (time 1783953453.11 total 1125.17 last 68.22) - open/close 26698 (time 1783953463.12 total 1135.18 last 72.04) - open/close 27337 (time 1783953473.13 total 1145.19 last 63.85) - open/close 28084 (time 1783953483.13 total 1155.19 last 74.65) - open/close 28937 (time 1783953493.13 total 1165.19 last 85.30) - open/close 29759 (time 1783953503.14 total 1175.20 last 82.14) - open/close 30000 (time 1783953505.79 total 1177.85 last 90.99) - open/close 30573 (time 1783953515.80 total 1187.86 last 57.24) - open/close 31157 (time 1783953525.82 total 1197.88 last 58.30) - open/close 31964 (time 1783953535.82 total 1207.88 last 80.69) - open/close 32594 (time 1783953545.82 total 1217.88 last 62.97) - open/close 33284 (time 1783953555.83 total 1227.89 last 68.95) - open/close 34091 (time 1783953565.84 total 1237.90 last 80.62) - open/close 34923 (time 1783953575.84 total 1247.90 last 83.18) - open/close 35723 (time 1783953585.87 total 1257.93 last 79.76) - open/close 36428 (time 1783953595.89 total 1267.95 last 70.41) - open/close 37266 (time 1783953605.90 total 1277.96 last 83.71) - open/close 38044 (time 1783953615.90 total 1287.96 last 77.78) - open/close 38883 (time 1783953625.91 total 1297.97 last 83.83) - open/close 39691 (time 1783953635.91 total 1307.97 last 80.76) - open/close 40000 (time 1783953639.92 total 1311.98 last 77.08) - open/close 40797 (time 1783953649.93 total 1321.99 last 79.65) - open/close 41563 (time 1783953659.94 total 1332.00 last 76.53) - open/close 42160 (time 1783953669.95 total 1342.01 last 59.66) - open/close 42677 (time 1783953679.98 total 1352.04 last 51.54) - open/close 43326 (time 1783953689.99 total 1362.05 last 64.82) - open/close 43856 (time 1783953699.99 total 1372.05 last 52.99) - open/close 44472 (time 1783953710.00 total 1382.06 last 61.52) - open/close 45197 (time 1783953720.02 total 1392.08 last 72.41) - open/close 45930 (time 1783953730.02 total 1402.08 last 73.27) - open/close 46616 (time 1783953740.03 total 1412.09 last 68.56) - open/close 47429 (time 1783953750.03 total 1422.09 last 81.30) - open/close 48135 (time 1783953760.03 total 1432.09 last 70.56) - open/close 48884 (time 1783953770.03 total 1442.09 last 74.90) - open/close 49602 (time 1783953780.03 total 1452.09 last 71.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1452.53 seconds: 34.17 ops/second - unlinked 0 (time 1783953785 ; total 0 ; last 0) - unlinked 10000 (time 1783953915 ; total 130 ; last 130) - unlinked 20000 (time 1783954062 ; total 277 ; last 147) - unlinked 30000 (time 1783954224 ; total 439 ; last 162) - unlinked 40000 (time 1783954371 ; total 586 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 721 seconds: 68.836342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2