== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:53:31 (1789088011) start mds service on oleg252-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg252-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg252-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg252-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 63 (time 1789088127.28 total 11.83 last 5.32) - open/close 319 (time 1789088138.18 total 22.74 last 23.48) - open/close 576 (time 1789088148.19 total 32.74 last 25.68) - open/close 732 (time 1789088158.32 total 42.88 last 15.39) - open/close 994 (time 1789088168.32 total 52.88 last 26.20) - open/close 1209 (time 1789088178.33 total 62.88 last 21.49) - open/close 1356 (time 1789088188.72 total 73.27 last 14.15) - open/close 1435 (time 1789088199.61 total 84.16 last 7.26) - open/close 1621 (time 1789088209.70 total 94.25 last 18.42) - open/close 1877 (time 1789088220.30 total 104.85 last 24.16) - open/close 2014 (time 1789088230.56 total 115.11 last 13.35) - open/close 2140 (time 1789088241.24 total 125.79 last 11.80) - open/close 2394 (time 1789088251.87 total 136.42 last 23.90) - open/close 2558 (time 1789088261.87 total 146.42 last 16.40) - open/close 2653 (time 1789088272.03 total 156.58 last 9.35) - open/close 2901 (time 1789088282.62 total 167.17 last 23.42) - open/close 3070 (time 1789088293.41 total 177.96 last 15.66) - open/close 3289 (time 1789088304.30 total 188.85 last 20.12) - open/close 3495 (time 1789088314.63 total 199.18 last 19.95) - open/close 3765 (time 1789088324.63 total 209.19 last 26.97) - open/close 3945 (time 1789088335.42 total 219.97 last 16.69) - open/close 4125 (time 1789088346.40 total 230.96 last 16.39) - open/close 4340 (time 1789088356.84 total 241.39 last 20.60) - open/close 4597 (time 1789088366.85 total 251.40 last 25.69) - open/close 4799 (time 1789088377.33 total 261.88 last 19.27) - open/close 4996 (time 1789088387.37 total 271.92 last 19.62) - open/close 5189 (time 1789088399.54 total 284.09 last 15.86) - open/close 5327 (time 1789088409.55 total 294.10 last 13.79) - open/close 5610 (time 1789088419.55 total 304.11 last 28.29) - open/close 5699 (time 1789088429.56 total 314.12 last 8.89) - open/close 5958 (time 1789088439.58 total 324.13 last 25.87) - open/close 6160 (time 1789088449.60 total 334.16 last 20.15) - open/close 6326 (time 1789088459.61 total 344.16 last 16.59) - open/close 6541 (time 1789088469.62 total 354.17 last 21.48) - open/close 6777 (time 1789088479.63 total 364.18 last 23.57) - open/close 7003 (time 1789088489.64 total 374.20 last 22.57) - open/close 7207 (time 1789088500.28 total 384.83 last 19.18) - open/close 7331 (time 1789088510.29 total 394.85 last 12.38) - open/close 7538 (time 1789088520.29 total 404.85 last 20.70) - open/close 7838 (time 1789088530.65 total 415.21 last 28.97) - open/close 8109 (time 1789088541.50 total 426.06 last 24.97) - open/close 8385 (time 1789088551.52 total 436.07 last 27.57) - open/close 8543 (time 1789088561.90 total 446.45 last 15.22) - open/close 8711 (time 1789088572.96 total 457.51 last 15.19) - open/close 9018 (time 1789088582.96 total 467.52 last 30.69) - open/close 9247 (time 1789088592.97 total 477.52 last 22.88) - open/close 9512 (time 1789088602.98 total 487.53 last 26.48) - open/close 9764 (time 1789088613.68 total 498.23 last 23.56) - open/close 10000 (time 1789088623.62 total 508.17 last 23.74) - open/close 10189 (time 1789088634.21 total 518.76 last 17.84) - open/close 10431 (time 1789088644.98 total 529.53 last 22.47) - open/close 10758 (time 1789088654.99 total 539.54 last 32.67) - open/close 10996 (time 1789088665.05 total 549.60 last 23.67) - open/close 11232 (time 1789088675.05 total 559.60 last 23.59) - open/close 11461 (time 1789088685.08 total 569.63 last 22.84) - open/close 11757 (time 1789088695.78 total 580.34 last 27.64) - open/close 11957 (time 1789088706.54 total 591.09 last 18.60) - open/close 12191 (time 1789088716.65 total 601.20 last 23.15) - open/close 12345 (time 1789088726.98 total 611.53 last 14.91) - open/close 12445 (time 1789088736.98 total 621.53 last 10.00) - open/close 12619 (time 1789088746.98 total 631.53 last 17.40) - open/close 12837 (time 1789088757.83 total 642.38 last 20.09) - open/close 12953 (time 1789088768.44 total 652.99 last 10.94) - open/close 13010 (time 1789088778.86 total 663.41 last 5.47) - open/close 13139 (time 1789088790.62 total 675.17 last 10.97) - open/close 13397 (time 1789088801.39 total 685.94 last 23.96) - open/close 13554 (time 1789088814.18 total 698.73 last 12.28) - open/close 13790 (time 1789088824.95 total 709.50 last 21.91) - open/close 14003 (time 1789088835.01 total 719.56 last 21.18) - open/close 14207 (time 1789088845.99 total 730.54 last 18.57) - open/close 14424 (time 1789088857.07 total 741.63 last 19.58) - open/close 14571 (time 1789088867.68 total 752.23 last 13.86) - open/close 14780 (time 1789088877.69 total 762.24 last 20.88) - open/close 15028 (time 1789088888.30 total 772.86 last 23.37) - open/close 15304 (time 1789088898.36 total 782.91 last 27.44) - open/close 15569 (time 1789088908.96 total 793.51 last 25.01) - open/close 15797 (time 1789088918.96 total 803.51 last 22.79) - open/close 15879 (time 1789088929.72 total 814.27 last 7.62) - open/close 16072 (time 1789088939.82 total 824.37 last 19.12) - open/close 16165 (time 1789088951.23 total 835.78 last 8.15) - open/close 16298 (time 1789088961.23 total 845.78 last 13.30) - open/close 16562 (time 1789088971.33 total 855.89 last 26.13) - open/close 16808 (time 1789088981.33 total 865.89 last 24.60) - open/close 16963 (time 1789088991.59 total 876.14 last 15.11) - open/close 17166 (time 1789089001.61 total 886.16 last 20.26) - open/close 17304 (time 1789089012.40 total 896.95 last 12.79) - open/close 17562 (time 1789089023.15 total 907.70 last 24.00) - open/close 17823 (time 1789089033.15 total 917.70 last 26.09) - open/close 18140 (time 1789089043.15 total 927.71 last 31.70) - open/close 18382 (time 1789089053.16 total 937.71 last 24.18) - open/close 18616 (time 1789089063.85 total 948.40 last 21.90) - open/close 18826 (time 1789089081.95 total 966.50 last 11.60) - open/close 19481 (time 1789089091.96 total 976.51 last 65.44) - open/close 20000 (time 1789089099.32 total 983.87 last 70.51) - open/close 20693 (time 1789089109.33 total 993.88 last 69.24) - open/close 21198 (time 1789089119.34 total 1003.90 last 50.42) - open/close 21856 (time 1789089129.35 total 1013.90 last 65.78) - open/close 22657 (time 1789089139.35 total 1023.90 last 80.08) - open/close 23647 (time 1789089149.36 total 1033.91 last 98.92) - open/close 24280 (time 1789089159.39 total 1043.95 last 63.08) - open/close 24962 (time 1789089169.39 total 1053.95 last 68.19) - open/close 25670 (time 1789089179.43 total 1063.98 last 70.55) - open/close 26368 (time 1789089189.44 total 1074.00 last 69.69) - open/close 27187 (time 1789089199.45 total 1084.00 last 81.88) - open/close 28000 (time 1789089209.46 total 1094.01 last 81.20) - open/close 28654 (time 1789089219.46 total 1104.01 last 65.39) - open/close 29371 (time 1789089229.49 total 1114.04 last 71.50) - open/close 30000 (time 1789089237.85 total 1122.40 last 75.27) - open/close 30806 (time 1789089247.85 total 1132.41 last 80.54) - open/close 31707 (time 1789089257.85 total 1142.41 last 90.09) - open/close 32538 (time 1789089267.86 total 1152.42 last 83.03) - open/close 33468 (time 1789089277.87 total 1162.42 last 92.93) - open/close 34253 (time 1789089287.88 total 1172.43 last 78.45) - open/close 35061 (time 1789089297.89 total 1182.44 last 80.73) - open/close 35901 (time 1789089307.89 total 1192.44 last 83.95) - open/close 36694 (time 1789089317.90 total 1202.45 last 79.27) - open/close 37505 (time 1789089327.91 total 1212.46 last 80.98) - open/close 38309 (time 1789089337.92 total 1222.47 last 80.35) - open/close 39178 (time 1789089347.92 total 1232.48 last 86.83) - open/close 40000 (time 1789089357.71 total 1242.26 last 84.04) - open/close 40855 (time 1789089367.71 total 1252.27 last 85.44) - open/close 41792 (time 1789089377.73 total 1262.28 last 93.58) - open/close 42674 (time 1789089387.73 total 1272.29 last 88.13) - open/close 43563 (time 1789089397.73 total 1282.29 last 88.90) - open/close 44482 (time 1789089407.74 total 1292.29 last 91.83) - open/close 45467 (time 1789089417.74 total 1302.30 last 98.48) - open/close 46439 (time 1789089427.75 total 1312.30 last 97.16) - open/close 47371 (time 1789089437.76 total 1322.31 last 93.11) - open/close 48127 (time 1789089447.76 total 1332.32 last 75.55) - open/close 48937 (time 1789089457.77 total 1342.33 last 80.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1349.92 seconds: 36.77 ops/second - unlinked 0 (time 1789089469 ; total 0 ; last 0) - unlinked 10000 (time 1789089607 ; total 138 ; last 138) - unlinked 20000 (time 1789089748 ; total 279 ; last 141) - unlinked 30000 (time 1789089878 ; total 409 ; last 130) - unlinked 40000 (time 1789090014 ; total 545 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 681 seconds: 72.879585 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: modules unloaded. pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2