== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:10:11 (1787573411) start mds service on oleg649-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg649-server' oleg649-server: oleg649-server.virtnet: executing load_modules_local oleg649-server: Loading modules from /home/green/git/lustre-release/lustre oleg649-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg649-server: detected 4 online CPUs by sysfs oleg649-server: MODOPTS_LIBCFS= oleg649-server: Force libcfs to create 2 CPU partitions oleg649-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg649-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg649-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg649-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg649-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg649-server: oleg649-server.virtnet: executing set_default_debug -1 all pdsh@oleg649-client: oleg649-server: ssh exited with exit code 1 Started lustre-OST0000 oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg649-client.virtnet: -o user_xattr,flock 192.168.206.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1787573508.72 total 10.61 last 23.36) - open/close 484 (time 1787573518.73 total 20.63 last 23.57) - open/close 717 (time 1787573529.19 total 31.08 last 22.28) - open/close 925 (time 1787573539.75 total 41.65 last 19.69) - open/close 1105 (time 1787573549.77 total 51.67 last 17.96) - open/close 1205 (time 1787573560.30 total 62.20 last 9.50) - open/close 1451 (time 1787573570.86 total 72.76 last 23.30) - open/close 1659 (time 1787573581.37 total 83.26 last 19.80) - open/close 1883 (time 1787573592.12 total 94.01 last 20.83) - open/close 2064 (time 1787573603.73 total 105.63 last 15.59) - open/close 2287 (time 1787573613.73 total 115.63 last 22.29) - open/close 2532 (time 1787573624.43 total 126.33 last 22.90) - open/close 2734 (time 1787573639.48 total 141.37 last 13.42) - open/close 3004 (time 1787573650.42 total 152.32 last 24.67) - open/close 3270 (time 1787573661.36 total 163.25 last 24.33) - open/close 3557 (time 1787573672.30 total 174.19 last 26.24) - open/close 3834 (time 1787573683.38 total 185.28 last 24.99) - open/close 4117 (time 1787573693.95 total 195.85 last 26.78) - open/close 4329 (time 1787573703.96 total 205.85 last 21.19) - open/close 4541 (time 1787573713.97 total 215.87 last 21.16) - open/close 4656 (time 1787573724.02 total 225.92 last 11.44) - open/close 4867 (time 1787573734.06 total 235.96 last 21.01) - open/close 5126 (time 1787573744.38 total 246.27 last 25.12) - open/close 5358 (time 1787573754.38 total 256.28 last 23.18) - open/close 5630 (time 1787573764.91 total 266.81 last 25.84) - open/close 5891 (time 1787573775.21 total 277.10 last 25.35) - open/close 6206 (time 1787573785.90 total 287.80 last 29.45) - open/close 6349 (time 1787573797.13 total 299.02 last 12.74) - open/close 6596 (time 1787573807.41 total 309.31 last 24.01) - open/close 6858 (time 1787573817.82 total 319.71 last 25.19) - open/close 7169 (time 1787573827.85 total 329.75 last 30.98) - open/close 7408 (time 1787573837.87 total 339.77 last 23.86) - open/close 7633 (time 1787573849.64 total 351.54 last 19.12) - open/close 7911 (time 1787573859.64 total 361.54 last 27.78) - open/close 8242 (time 1787573869.65 total 371.55 last 33.09) - open/close 8501 (time 1787573879.65 total 381.55 last 25.89) - open/close 8750 (time 1787573889.67 total 391.57 last 24.85) - open/close 9010 (time 1787573899.69 total 401.59 last 25.96) - open/close 9327 (time 1787573910.58 total 412.48 last 29.10) - open/close 9644 (time 1787573920.89 total 422.78 last 30.77) - open/close 9944 (time 1787573931.83 total 433.73 last 27.41) - open/close 10251 (time 1787573942.87 total 444.76 last 27.82) - open/close 10529 (time 1787573953.17 total 455.07 last 26.97) - open/close 10737 (time 1787573963.98 total 465.87 last 19.25) - open/close 11024 (time 1787573976.90 total 478.80 last 22.21) - open/close 11269 (time 1787573986.95 total 488.85 last 24.37) - open/close 11391 (time 1787573997.35 total 499.25 last 11.73) - open/close 11613 (time 1787574007.98 total 509.88 last 20.89) - open/close 11921 (time 1787574019.04 total 520.94 last 27.84) - open/close 12224 (time 1787574029.05 total 530.95 last 30.28) - open/close 12443 (time 1787574039.40 total 541.29 last 21.16) - open/close 12685 (time 1787574050.22 total 552.11 last 22.37) - open/close 12911 (time 1787574060.99 total 562.89 last 20.98) - open/close 13167 (time 1787574071.08 total 572.98 last 25.36) - open/close 13441 (time 1787574081.13 total 583.03 last 27.27) - open/close 13748 (time 1787574092.03 total 593.92 last 28.18) - open/close 14009 (time 1787574103.86 total 605.76 last 22.05) - open/close 14271 (time 1787574114.34 total 616.23 last 25.01) - open/close 14477 (time 1787574125.29 total 627.19 last 18.81) - open/close 14622 (time 1787574135.35 total 637.25 last 14.41) - open/close 14745 (time 1787574145.85 total 647.75 last 11.71) - open/close 14953 (time 1787574155.85 total 657.75 last 20.80) - open/close 15218 (time 1787574165.86 total 667.75 last 26.49) - open/close 15495 (time 1787574175.88 total 677.78 last 27.63) - open/close 15732 (time 1787574186.08 total 687.98 last 23.23) - open/close 16032 (time 1787574196.52 total 698.42 last 28.75) - open/close 16303 (time 1787574206.90 total 708.80 last 26.11) - open/close 16557 (time 1787574216.93 total 718.83 last 25.32) - open/close 16829 (time 1787574227.25 total 729.15 last 26.36) - open/close 17091 (time 1787574237.56 total 739.46 last 25.41) - open/close 17227 (time 1787574247.59 total 749.49 last 13.56) - open/close 17264 (time 1787574257.88 total 759.77 last 3.60) - open/close 17341 (time 1787574267.88 total 769.77 last 7.70) - open/close 17495 (time 1787574278.71 total 780.60 last 14.22) - open/close 17636 (time 1787574289.33 total 791.23 last 13.27) - open/close 17906 (time 1787574299.34 total 801.24 last 26.98) - open/close 18158 (time 1787574309.37 total 811.27 last 25.12) - open/close 18415 (time 1787574320.27 total 822.16 last 23.58) - open/close 18687 (time 1787574330.62 total 832.51 last 26.28) - open/close 18951 (time 1787574341.29 total 843.18 last 24.75) - open/close 19165 (time 1787574351.29 total 853.18 last 21.40) - open/close 19416 (time 1787574361.51 total 863.41 last 24.55) - open/close 19673 (time 1787574371.51 total 873.41 last 25.70) - open/close 19937 (time 1787574381.52 total 883.41 last 26.38) - open/close 20000 (time 1787574384.79 total 886.69 last 19.23) - open/close 20303 (time 1787574395.64 total 897.54 last 27.94) - open/close 20561 (time 1787574405.81 total 907.70 last 25.37) - open/close 21209 (time 1787574415.83 total 917.73 last 64.65) - open/close 22178 (time 1787574425.84 total 927.73 last 96.84) - open/close 23185 (time 1787574435.85 total 937.74 last 100.60) - open/close 24161 (time 1787574445.86 total 947.76 last 97.47) - open/close 25069 (time 1787574455.86 total 957.76 last 90.79) - open/close 26024 (time 1787574465.86 total 967.76 last 95.50) - open/close 26873 (time 1787574475.88 total 977.77 last 84.79) - open/close 27688 (time 1787574485.88 total 987.78 last 81.45) - open/close 28581 (time 1787574495.89 total 997.78 last 89.24) - open/close 29530 (time 1787574505.89 total 1007.78 last 94.89) - open/close 30000 (time 1787574511.77 total 1013.66 last 79.96) - open/close 30880 (time 1787574521.79 total 1023.69 last 87.79) - open/close 31806 (time 1787574531.81 total 1033.70 last 92.45) - open/close 32746 (time 1787574541.81 total 1043.71 last 93.96) - open/close 33696 (time 1787574551.81 total 1053.71 last 94.95) - open/close 34534 (time 1787574561.84 total 1063.74 last 83.60) - open/close 35395 (time 1787574571.84 total 1073.74 last 86.05) - open/close 36123 (time 1787574581.85 total 1083.75 last 72.73) - open/close 37082 (time 1787574591.86 total 1093.76 last 95.83) - open/close 38072 (time 1787574601.87 total 1103.77 last 98.90) - open/close 38972 (time 1787574611.88 total 1113.77 last 89.94) - open/close 39786 (time 1787574621.88 total 1123.78 last 81.36) - open/close 40605 (time 1787574631.90 total 1133.79 last 81.79) - open/close 41461 (time 1787574641.90 total 1143.79 last 85.59) - open/close 42356 (time 1787574651.91 total 1153.80 last 89.42) - open/close 43250 (time 1787574661.91 total 1163.80 last 89.39) - open/close 44043 (time 1787574671.91 total 1173.81 last 79.24) - open/close 44970 (time 1787574681.92 total 1183.82 last 92.63) - open/close 45913 (time 1787574691.92 total 1193.82 last 94.29) - open/close 46797 (time 1787574701.93 total 1203.82 last 88.37) - open/close 47665 (time 1787574711.93 total 1213.83 last 86.78) - open/close 48401 (time 1787574721.93 total 1223.83 last 73.58) - open/close 49059 (time 1787574731.94 total 1233.84 last 65.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1241.05 seconds: 39.99 ops/second - unlinked 0 (time 1787574744 ; total 0 ; last 0) - unlinked 10000 (time 1787574864 ; total 120 ; last 120) - unlinked 20000 (time 1787574967 ; total 223 ; last 103) - unlinked 30000 (time 1787575028 ; total 284 ; last 61) - unlinked 40000 (time 1787575091 ; total 347 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 407 seconds: 121.943489 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg649-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg649-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg649-server stop mds service on oleg649-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg649-server stop mds service on oleg649-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg649-server unloading modules via unload_modules_local on: 'oleg649-server' oleg649-server: oleg649-server.virtnet: executing unload_modules_local oleg649-server: modules unloaded. pdsh@oleg649-client: oleg649-client: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-server: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-client: ssh exited with exit code 2 pdsh@oleg649-client: oleg649-server: ssh exited with exit code 2