== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:37:46 (1788446266) start mds service on oleg205-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions oleg205-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg205-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg205-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg205-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1788446381.47 total 10.01 last 16.58) - open/close 300 (time 1788446391.89 total 20.44 last 12.86) - open/close 461 (time 1788446404.19 total 32.74 last 13.09) - open/close 574 (time 1788446414.55 total 43.10 last 10.91) - open/close 762 (time 1788446424.75 total 53.30 last 18.43) - open/close 1010 (time 1788446434.76 total 63.31 last 24.78) - open/close 1200 (time 1788446444.78 total 73.32 last 18.97) - open/close 1355 (time 1788446456.25 total 84.80 last 13.51) - open/close 1582 (time 1788446466.46 total 95.00 last 22.25) - open/close 1713 (time 1788446476.46 total 105.01 last 13.10) - open/close 1931 (time 1788446486.96 total 115.51 last 20.76) - open/close 2180 (time 1788446496.98 total 125.52 last 24.86) - open/close 2338 (time 1788446507.66 total 136.21 last 14.79) - open/close 2459 (time 1788446518.49 total 147.04 last 11.17) - open/close 2636 (time 1788446529.18 total 157.72 last 16.56) - open/close 2853 (time 1788446539.81 total 168.35 last 20.42) - open/close 3061 (time 1788446549.81 total 178.35 last 20.79) - open/close 3350 (time 1788446560.49 total 189.03 last 27.06) - open/close 3479 (time 1788446571.54 total 200.09 last 11.67) - open/close 3644 (time 1788446583.95 total 212.50 last 13.29) - open/close 3826 (time 1788446593.96 total 222.51 last 18.19) - open/close 3991 (time 1788446605.65 total 234.20 last 14.11) - open/close 4267 (time 1788446616.06 total 244.60 last 26.53) - open/close 4475 (time 1788446626.25 total 254.79 last 20.42) - open/close 4643 (time 1788446636.78 total 265.32 last 15.95) - open/close 4907 (time 1788446647.80 total 276.35 last 23.95) - open/close 5186 (time 1788446658.34 total 286.88 last 26.48) - open/close 5429 (time 1788446668.34 total 296.89 last 24.29) - open/close 5581 (time 1788446679.02 total 307.57 last 14.23) - open/close 5705 (time 1788446689.03 total 317.58 last 12.39) - open/close 5790 (time 1788446699.04 total 327.58 last 8.49) - open/close 5956 (time 1788446709.49 total 338.03 last 15.89) - open/close 6130 (time 1788446719.93 total 348.47 last 16.66) - open/close 6271 (time 1788446730.10 total 358.65 last 13.86) - open/close 6481 (time 1788446740.43 total 368.98 last 20.33) - open/close 6666 (time 1788446750.45 total 378.99 last 18.47) - open/close 6917 (time 1788446761.47 total 390.01 last 22.78) - open/close 7106 (time 1788446771.47 total 400.01 last 18.90) - open/close 7342 (time 1788446781.73 total 410.27 last 23.00) - open/close 7562 (time 1788446791.76 total 420.30 last 21.93) - open/close 7734 (time 1788446802.40 total 430.95 last 16.16) - open/close 7918 (time 1788446813.16 total 441.70 last 17.11) - open/close 8110 (time 1788446823.69 total 452.24 last 18.22) - open/close 8274 (time 1788446834.34 total 462.89 last 15.40) - open/close 8505 (time 1788446844.34 total 472.89 last 23.10) - open/close 8712 (time 1788446854.36 total 482.90 last 20.67) - open/close 8852 (time 1788446864.43 total 492.97 last 13.90) - open/close 9150 (time 1788446874.66 total 503.20 last 29.13) - open/close 9458 (time 1788446887.38 total 515.92 last 24.22) - open/close 9699 (time 1788446898.14 total 526.68 last 22.40) - open/close 9940 (time 1788446908.15 total 536.70 last 24.07) - open/close 10000 (time 1788446911.47 total 540.01 last 18.11) - open/close 10249 (time 1788446922.07 total 550.61 last 23.48) - open/close 10447 (time 1788446932.11 total 560.66 last 19.72) - open/close 10708 (time 1788446942.23 total 570.77 last 25.80) - open/close 10942 (time 1788446952.24 total 580.78 last 23.38) - open/close 11216 (time 1788446963.29 total 591.83 last 24.80) - open/close 11315 (time 1788446974.05 total 602.59 last 9.20) - open/close 11461 (time 1788446984.21 total 612.76 last 14.36) - open/close 11717 (time 1788446994.86 total 623.40 last 24.05) - open/close 11942 (time 1788447004.88 total 633.43 last 22.45) - open/close 12200 (time 1788447015.98 total 644.53 last 23.24) - open/close 12382 (time 1788447026.90 total 655.45 last 16.67) - open/close 12571 (time 1788447036.92 total 665.47 last 18.86) - open/close 12751 (time 1788447051.69 total 680.24 last 12.19) - open/close 12988 (time 1788447062.43 total 690.98 last 22.07) - open/close 13211 (time 1788447072.93 total 701.47 last 21.25) - open/close 13459 (time 1788447082.93 total 711.47 last 24.80) - open/close 13611 (time 1788447094.68 total 723.23 last 12.93) - open/close 13802 (time 1788447105.57 total 734.12 last 17.54) - open/close 13910 (time 1788447117.39 total 745.94 last 9.13) - open/close 14051 (time 1788447127.41 total 755.96 last 14.08) - open/close 14231 (time 1788447137.95 total 766.49 last 17.08) - open/close 14427 (time 1788447147.96 total 776.50 last 19.58) - open/close 14592 (time 1788447157.96 total 786.50 last 16.49) - open/close 14884 (time 1788447167.97 total 796.51 last 29.17) - open/close 15058 (time 1788447177.99 total 806.53 last 17.37) - open/close 15307 (time 1788447188.00 total 816.54 last 24.87) - open/close 15509 (time 1788447198.66 total 827.20 last 18.95) - open/close 15688 (time 1788447210.66 total 839.20 last 14.92) - open/close 15792 (time 1788447221.43 total 849.97 last 9.66) - open/close 16071 (time 1788447231.45 total 859.99 last 27.86) - open/close 16319 (time 1788447242.14 total 870.68 last 23.19) - open/close 16603 (time 1788447252.14 total 880.69 last 28.39) - open/close 16830 (time 1788447262.15 total 890.69 last 22.69) - open/close 17053 (time 1788447272.32 total 900.86 last 21.92) - open/close 17235 (time 1788447282.33 total 910.88 last 18.17) - open/close 17508 (time 1788447292.35 total 920.90 last 27.25) - open/close 17702 (time 1788447303.34 total 931.88 last 17.65) - open/close 17904 (time 1788447314.20 total 942.75 last 18.60) - open/close 18096 (time 1788447324.21 total 952.76 last 19.18) - open/close 18312 (time 1788447334.21 total 962.76 last 21.60) - open/close 18492 (time 1788447344.22 total 972.77 last 17.98) - open/close 18613 (time 1788447354.28 total 982.83 last 12.03) - open/close 18739 (time 1788447367.33 total 995.88 last 9.66) - open/close 18868 (time 1788447378.20 total 1006.74 last 11.87) - open/close 19137 (time 1788447388.21 total 1016.75 last 26.87) - open/close 19921 (time 1788447398.21 total 1026.76 last 78.36) - open/close 20766 (time 1788447408.23 total 1036.77 last 84.38) - open/close 21487 (time 1788447418.24 total 1046.79 last 71.96) - open/close 22156 (time 1788447428.26 total 1056.80 last 66.82) - open/close 22858 (time 1788447438.26 total 1066.81 last 70.17) - open/close 23695 (time 1788447448.27 total 1076.81 last 83.64) - open/close 24469 (time 1788447458.28 total 1086.83 last 77.30) - open/close 25210 (time 1788447468.29 total 1096.83 last 74.05) - open/close 25984 (time 1788447478.29 total 1106.83 last 77.39) - open/close 26774 (time 1788447488.31 total 1116.85 last 78.84) - open/close 27658 (time 1788447498.31 total 1126.85 last 88.40) - open/close 28443 (time 1788447508.31 total 1136.86 last 78.49) - open/close 29182 (time 1788447518.32 total 1146.86 last 73.87) - open/close 30000 (time 1788447527.92 total 1156.46 last 85.21) - open/close 30783 (time 1788447537.92 total 1166.46 last 78.30) - open/close 31467 (time 1788447547.92 total 1176.46 last 68.40) - open/close 32034 (time 1788447557.93 total 1186.47 last 56.65) - open/close 32554 (time 1788447567.93 total 1196.47 last 51.99) - open/close 33245 (time 1788447577.93 total 1206.48 last 69.05) - open/close 34007 (time 1788447587.94 total 1216.49 last 76.13) - open/close 34767 (time 1788447597.96 total 1226.50 last 75.90) - open/close 35435 (time 1788447607.98 total 1236.52 last 66.64) - open/close 35920 (time 1788447617.99 total 1246.54 last 48.44) - open/close 36446 (time 1788447628.00 total 1256.54 last 52.57) - open/close 37013 (time 1788447638.01 total 1266.55 last 56.65) - open/close 37654 (time 1788447648.02 total 1276.56 last 64.03) - open/close 38342 (time 1788447658.02 total 1286.56 last 68.78) - open/close 38897 (time 1788447668.05 total 1296.59 last 55.36) - open/close 39380 (time 1788447678.05 total 1306.60 last 48.27) - open/close 40000 (time 1788447685.68 total 1314.23 last 81.23) - open/close 40656 (time 1788447695.74 total 1324.29 last 65.20) - open/close 41392 (time 1788447705.75 total 1334.29 last 73.58) - open/close 42257 (time 1788447715.75 total 1344.29 last 86.48) - open/close 42971 (time 1788447725.76 total 1354.31 last 71.31) - open/close 43865 (time 1788447735.76 total 1364.31 last 89.38) - open/close 44544 (time 1788447745.79 total 1374.33 last 67.75) - open/close 45330 (time 1788447755.79 total 1384.34 last 78.54) - open/close 46110 (time 1788447765.81 total 1394.35 last 77.92) - open/close 46821 (time 1788447775.82 total 1404.37 last 70.98) - open/close 47604 (time 1788447785.83 total 1414.37 last 78.28) - open/close 48306 (time 1788447795.84 total 1424.39 last 70.08) - open/close 49066 (time 1788447805.84 total 1434.39 last 75.99) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1441.83 seconds: 34.42 ops/second - unlinked 0 (time 1788447819 ; total 0 ; last 0) - unlinked 10000 (time 1788447971 ; total 152 ; last 152) - unlinked 20000 (time 1788448116 ; total 297 ; last 145) - unlinked 30000 (time 1788448255 ; total 436 ; last 139) - unlinked 40000 (time 1788448391 ; total 572 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 701 seconds: 70.800285 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: modules unloaded. pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2