== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:40 (1787579980) start mds service on oleg131-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1787580096.32 total 10.36 last 19.99) - open/close 462 (time 1787580108.27 total 22.31 last 21.34) - open/close 704 (time 1787580118.69 total 32.73 last 23.21) - open/close 846 (time 1787580128.88 total 42.92 last 13.94) - open/close 977 (time 1787580138.90 total 52.94 last 13.07) - open/close 1212 (time 1787580148.91 total 62.95 last 23.47) - open/close 1395 (time 1787580158.92 total 72.96 last 18.29) - open/close 1577 (time 1787580168.93 total 82.97 last 18.17) - open/close 1781 (time 1787580179.17 total 93.21 last 19.94) - open/close 2005 (time 1787580189.62 total 103.66 last 21.42) - open/close 2191 (time 1787580199.63 total 113.67 last 18.59) - open/close 2382 (time 1787580209.63 total 123.67 last 19.10) - open/close 2563 (time 1787580219.83 total 133.87 last 17.75) - open/close 2801 (time 1787580231.42 total 145.46 last 20.54) - open/close 2961 (time 1787580244.90 total 158.94 last 11.87) - open/close 3224 (time 1787580255.02 total 169.06 last 25.98) - open/close 3386 (time 1787580265.05 total 179.09 last 16.15) - open/close 3564 (time 1787580275.17 total 189.21 last 17.60) - open/close 3636 (time 1787580285.19 total 199.23 last 7.18) - open/close 3701 (time 1787580295.21 total 209.25 last 6.48) - open/close 3934 (time 1787580305.21 total 219.25 last 23.30) - open/close 4112 (time 1787580315.81 total 229.85 last 16.80) - open/close 4373 (time 1787580325.82 total 239.86 last 26.08) - open/close 4482 (time 1787580337.46 total 251.50 last 9.36) - open/close 4737 (time 1787580347.49 total 261.53 last 25.42) - open/close 4999 (time 1787580357.68 total 271.72 last 25.71) - open/close 5086 (time 1787580368.12 total 282.16 last 8.33) - open/close 5328 (time 1787580378.92 total 292.96 last 22.39) - open/close 5583 (time 1787580388.94 total 302.98 last 25.47) - open/close 5830 (time 1787580399.73 total 313.77 last 22.89) - open/close 5910 (time 1787580410.61 total 324.65 last 7.35) - open/close 6021 (time 1787580421.05 total 335.09 last 10.64) - open/close 6153 (time 1787580432.10 total 346.14 last 11.94) - open/close 6325 (time 1787580443.18 total 357.22 last 15.52) - open/close 6451 (time 1787580453.19 total 367.23 last 12.59) - open/close 6639 (time 1787580463.21 total 377.25 last 18.76) - open/close 6771 (time 1787580473.23 total 387.27 last 13.17) - open/close 6916 (time 1787580483.24 total 397.28 last 14.48) - open/close 7084 (time 1787580493.67 total 407.71 last 16.12) - open/close 7267 (time 1787580504.61 total 418.65 last 16.73) - open/close 7388 (time 1787580514.66 total 428.70 last 12.03) - open/close 7559 (time 1787580525.28 total 439.32 last 16.11) - open/close 7814 (time 1787580535.86 total 449.90 last 24.10) - open/close 7981 (time 1787580546.55 total 460.59 last 15.62) - open/close 8063 (time 1787580556.98 total 471.02 last 7.86) - open/close 8184 (time 1787580568.05 total 482.09 last 10.93) - open/close 8314 (time 1787580578.12 total 492.15 last 12.91) - open/close 8426 (time 1787580588.81 total 502.85 last 10.47) - open/close 8677 (time 1787580598.88 total 512.92 last 24.92) - open/close 8802 (time 1787580608.88 total 522.92 last 12.49) - open/close 8919 (time 1787580618.89 total 532.93 last 11.70) - open/close 9193 (time 1787580628.92 total 542.96 last 27.31) - open/close 9402 (time 1787580638.92 total 552.96 last 20.89) - open/close 9619 (time 1787580648.94 total 562.98 last 21.67) - open/close 9836 (time 1787580658.98 total 573.02 last 21.61) - open/close 10000 (time 1787580665.78 total 579.81 last 24.12) - open/close 10259 (time 1787580676.72 total 590.76 last 23.66) - open/close 10474 (time 1787580687.21 total 601.25 last 20.51) - open/close 10590 (time 1787580697.22 total 611.25 last 11.59) - open/close 10681 (time 1787580707.22 total 621.26 last 9.09) - open/close 10868 (time 1787580717.24 total 631.28 last 18.67) - open/close 11090 (time 1787580728.18 total 642.22 last 20.29) - open/close 11288 (time 1787580738.19 total 652.23 last 19.79) - open/close 11570 (time 1787580748.78 total 662.82 last 26.62) - open/close 11805 (time 1787580759.40 total 673.44 last 22.12) - open/close 11935 (time 1787580769.41 total 683.45 last 13.00) - open/close 12186 (time 1787580779.42 total 693.46 last 25.07) - open/close 12304 (time 1787580790.00 total 704.04 last 11.15) - open/close 12504 (time 1787580802.50 total 716.54 last 16.01) - open/close 12745 (time 1787580812.97 total 727.01 last 23.02) - open/close 12906 (time 1787580823.21 total 737.25 last 15.71) - open/close 13160 (time 1787580833.45 total 747.49 last 24.82) - open/close 13361 (time 1787580843.95 total 757.99 last 19.14) - open/close 13544 (time 1787580854.31 total 768.35 last 17.66) - open/close 13684 (time 1787580865.19 total 779.22 last 12.88) - open/close 13716 (time 1787580875.27 total 789.30 last 3.17) - open/close 13972 (time 1787580885.29 total 799.33 last 25.53) - open/close 14166 (time 1787580895.31 total 809.35 last 19.36) - open/close 14371 (time 1787580905.46 total 819.50 last 20.21) - open/close 14611 (time 1787580915.54 total 829.58 last 23.80) - open/close 14858 (time 1787580927.48 total 841.52 last 20.69) - open/close 15018 (time 1787580937.48 total 851.52 last 15.99) - open/close 15249 (time 1787580947.63 total 861.67 last 22.76) - open/close 15467 (time 1787580957.64 total 871.67 last 21.79) - open/close 15709 (time 1787580967.67 total 881.71 last 24.12) - open/close 15808 (time 1787580977.68 total 891.72 last 9.88) - open/close 15983 (time 1787580988.16 total 902.20 last 16.71) - open/close 16152 (time 1787580998.55 total 912.59 last 16.25) - open/close 16272 (time 1787581008.56 total 922.60 last 11.99) - open/close 16507 (time 1787581021.62 total 935.66 last 17.99) - open/close 16627 (time 1787581031.80 total 945.84 last 11.80) - open/close 16787 (time 1787581042.16 total 956.20 last 15.45) - open/close 16887 (time 1787581052.91 total 966.95 last 9.30) - open/close 16952 (time 1787581062.91 total 976.95 last 6.50) - open/close 17147 (time 1787581073.46 total 987.50 last 18.48) - open/close 17281 (time 1787581083.82 total 997.86 last 12.93) - open/close 17498 (time 1787581093.83 total 1007.87 last 21.69) - open/close 17627 (time 1787581104.89 total 1018.93 last 11.66) - open/close 17698 (time 1787581114.90 total 1028.94 last 7.10) - open/close 17837 (time 1787581124.93 total 1038.97 last 13.86) - open/close 17961 (time 1787581134.96 total 1049.00 last 12.36) - open/close 18596 (time 1787581144.98 total 1059.02 last 63.37) - open/close 19296 (time 1787581155.00 total 1069.04 last 69.90) - open/close 20000 (time 1787581163.38 total 1077.42 last 83.99) - open/close 20777 (time 1787581173.40 total 1087.44 last 77.56) - open/close 21574 (time 1787581183.40 total 1097.44 last 79.70) - open/close 22608 (time 1787581193.40 total 1107.44 last 103.37) - open/close 23511 (time 1787581203.40 total 1117.44 last 90.28) - open/close 24402 (time 1787581213.40 total 1127.44 last 89.09) - open/close 25364 (time 1787581223.41 total 1137.45 last 96.17) - open/close 26318 (time 1787581233.41 total 1147.45 last 95.35) - open/close 27380 (time 1787581243.42 total 1157.46 last 106.14) - open/close 28212 (time 1787581253.42 total 1167.46 last 83.16) - open/close 29082 (time 1787581263.44 total 1177.48 last 86.89) - open/close 29955 (time 1787581273.45 total 1187.48 last 87.22) - open/close 30737 (time 1787581283.45 total 1197.49 last 78.16) - open/close 31633 (time 1787581293.45 total 1207.49 last 89.58) - open/close 32623 (time 1787581303.46 total 1217.50 last 98.93) - open/close 33403 (time 1787581313.47 total 1227.51 last 77.95) - open/close 34050 (time 1787581323.47 total 1237.51 last 64.67) - open/close 34730 (time 1787581333.48 total 1247.52 last 67.96) - open/close 35503 (time 1787581343.48 total 1257.52 last 77.27) - open/close 36240 (time 1787581353.49 total 1267.53 last 73.64) - open/close 36947 (time 1787581363.49 total 1277.53 last 70.68) - open/close 37747 (time 1787581373.50 total 1287.54 last 79.94) - open/close 38460 (time 1787581383.50 total 1297.54 last 71.25) - open/close 39255 (time 1787581393.51 total 1307.55 last 79.46) - open/close 39935 (time 1787581403.52 total 1317.56 last 67.95) - open/close 40602 (time 1787581413.53 total 1327.57 last 66.58) - open/close 41180 (time 1787581423.54 total 1337.58 last 57.76) - open/close 41630 (time 1787581433.56 total 1347.60 last 44.93) - open/close 42196 (time 1787581443.57 total 1357.61 last 56.55) - open/close 42667 (time 1787581453.58 total 1367.62 last 47.05) - open/close 43229 (time 1787581463.58 total 1377.62 last 56.18) - open/close 44004 (time 1787581473.58 total 1387.62 last 77.48) - open/close 45032 (time 1787581483.59 total 1397.63 last 102.74) - open/close 46035 (time 1787581493.60 total 1407.64 last 100.17) - open/close 47113 (time 1787581503.61 total 1417.65 last 107.74) - open/close 47932 (time 1787581513.61 total 1427.65 last 81.88) - open/close 48768 (time 1787581523.63 total 1437.67 last 83.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1447.28 seconds: 34.29 ops/second - unlinked 0 (time 1787581537 ; total 0 ; last 0) - unlinked 10000 (time 1787581643 ; total 106 ; last 106) - unlinked 20000 (time 1787581744 ; total 207 ; last 101) - unlinked 30000 (time 1787581816 ; total 279 ; last 72) - unlinked 40000 (time 1787581873 ; total 336 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 388 seconds: 127.914948 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2