== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:43:42 (1782614622) start mds service on oleg149-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing load_modules_local oleg149-server: Loading modules from /home/green/git/lustre-release/lustre oleg149-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg149-server: detected 4 online CPUs by sysfs oleg149-server: MODOPTS_LIBCFS= oleg149-server: Force libcfs to create 2 CPU partitions oleg149-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg149-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg149-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg149-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg149-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg149-client.virtnet: -o user_xattr,flock 192.168.201.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1782614740.47 total 10.85 last 20.74) - open/close 467 (time 1782614750.48 total 20.86 last 24.17) - open/close 704 (time 1782614760.79 total 31.17 last 22.98) - open/close 894 (time 1782614770.80 total 41.18 last 18.97) - open/close 1057 (time 1782614781.30 total 51.68 last 15.53) - open/close 1137 (time 1782614791.61 total 61.99 last 7.75) - open/close 1357 (time 1782614801.62 total 72.00 last 21.99) - open/close 1559 (time 1782614811.62 total 82.00 last 20.19) - open/close 1785 (time 1782614821.70 total 92.08 last 22.43) - open/close 1923 (time 1782614831.80 total 102.18 last 13.66) - open/close 2073 (time 1782614842.29 total 112.67 last 14.29) - open/close 2246 (time 1782614852.31 total 122.69 last 17.26) - open/close 2372 (time 1782614863.30 total 133.68 last 11.47) - open/close 2574 (time 1782614873.31 total 143.69 last 20.17) - open/close 2725 (time 1782614883.64 total 154.02 last 14.63) - open/close 2915 (time 1782614893.81 total 164.19 last 18.68) - open/close 3018 (time 1782614904.44 total 174.82 last 9.69) - open/close 3196 (time 1782614914.95 total 185.33 last 16.93) - open/close 3311 (time 1782614925.11 total 195.49 last 11.32) - open/close 3419 (time 1782614935.12 total 205.50 last 10.78) - open/close 3565 (time 1782614945.27 total 215.65 last 14.39) - open/close 3756 (time 1782614955.71 total 226.09 last 18.29) - open/close 3888 (time 1782614965.71 total 236.09 last 13.19) - open/close 4039 (time 1782614975.74 total 246.12 last 15.06) - open/close 4216 (time 1782614990.74 total 261.12 last 11.80) - open/close 4396 (time 1782615000.78 total 271.16 last 17.92) - open/close 4544 (time 1782615011.31 total 281.69 last 14.05) - open/close 4751 (time 1782615021.33 total 291.71 last 20.67) - open/close 4931 (time 1782615032.30 total 302.68 last 16.41) - open/close 5153 (time 1782615043.52 total 313.90 last 19.78) - open/close 5312 (time 1782615053.55 total 323.93 last 15.86) - open/close 5389 (time 1782615064.13 total 334.51 last 7.28) - open/close 5512 (time 1782615074.13 total 344.51 last 12.29) - open/close 5582 (time 1782615085.16 total 355.54 last 6.34) - open/close 5801 (time 1782615095.74 total 366.12 last 20.70) - open/close 5949 (time 1782615106.16 total 376.54 last 14.20) - open/close 6122 (time 1782615118.21 total 388.59 last 14.36) - open/close 6246 (time 1782615128.64 total 399.02 last 11.88) - open/close 6397 (time 1782615138.65 total 409.03 last 15.09) - open/close 6487 (time 1782615151.43 total 421.81 last 7.04) - open/close 6591 (time 1782615161.44 total 431.82 last 10.39) - open/close 6717 (time 1782615174.07 total 444.45 last 9.98) - open/close 6871 (time 1782615184.21 total 454.59 last 15.18) - open/close 6952 (time 1782615195.12 total 465.50 last 7.42) - open/close 6992 (time 1782615205.38 total 475.76 last 3.90) - open/close 7156 (time 1782615215.39 total 485.77 last 16.38) - open/close 7237 (time 1782615225.85 total 496.23 last 7.74) - open/close 7344 (time 1782615235.90 total 506.28 last 10.65) - open/close 7585 (time 1782615245.90 total 516.29 last 24.08) - open/close 7651 (time 1782615256.69 total 527.07 last 6.12) - open/close 7740 (time 1782615267.14 total 537.52 last 8.52) - open/close 7844 (time 1782615277.15 total 547.53 last 10.40) - open/close 8070 (time 1782615287.87 total 558.25 last 21.07) - open/close 8331 (time 1782615300.93 total 571.31 last 19.99) - open/close 8444 (time 1782615311.64 total 582.02 last 10.55) - open/close 8596 (time 1782615321.97 total 592.35 last 14.71) - open/close 8667 (time 1782615332.10 total 602.48 last 7.01) - open/close 8754 (time 1782615344.36 total 614.74 last 7.10) - open/close 8864 (time 1782615355.04 total 625.42 last 10.30) - open/close 9000 (time 1782615365.51 total 635.89 last 12.98) - open/close 9132 (time 1782615377.87 total 648.25 last 10.68) - open/close 9344 (time 1782615388.92 total 659.30 last 19.18) - open/close 9454 (time 1782615401.77 total 672.15 last 8.56) - open/close 9584 (time 1782615412.81 total 683.19 last 11.78) - open/close 9674 (time 1782615423.61 total 693.99 last 8.33) - open/close 9789 (time 1782615434.04 total 704.42 last 11.02) - open/close 9988 (time 1782615444.34 total 714.72 last 19.32) - open/close 10108 (time 1782615454.35 total 724.73 last 11.99) - open/close 10335 (time 1782615464.86 total 735.24 last 21.61) - open/close 10463 (time 1782615475.67 total 746.05 last 11.84) - open/close 10655 (time 1782615488.46 total 758.84 last 15.01) - open/close 10814 (time 1782615498.95 total 769.33 last 15.16) - open/close 10987 (time 1782615509.18 total 779.57 last 16.90) - open/close 11086 (time 1782615522.08 total 792.46 last 7.68) - open/close 11157 (time 1782615532.47 total 802.85 last 6.83) - open/close 11313 (time 1782615543.51 total 813.89 last 14.14) - open/close 11401 (time 1782615555.95 total 826.33 last 7.07) - open/close 11499 (time 1782615566.78 total 837.16 last 9.05) - open/close 11558 (time 1782615577.52 total 847.90 last 5.49) - open/close 11762 (time 1782615588.42 total 858.80 last 18.72) - open/close 11965 (time 1782615599.15 total 869.53 last 18.92) - open/close 12069 (time 1782615609.79 total 880.17 last 9.77) - open/close 12255 (time 1782615619.80 total 890.18 last 18.59) - open/close 12321 (time 1782615629.82 total 900.20 last 6.59) - open/close 12394 (time 1782615640.21 total 910.59 last 7.03) - open/close 12473 (time 1782615650.75 total 921.14 last 7.49) - open/close 12620 (time 1782615661.29 total 931.68 last 13.95) - open/close 12805 (time 1782615671.42 total 941.80 last 18.28) - open/close 13015 (time 1782615681.93 total 952.31 last 19.98) - open/close 13206 (time 1782615691.94 total 962.32 last 19.08) - open/close 13281 (time 1782615703.20 total 973.58 last 6.66) - open/close 13449 (time 1782615713.21 total 983.59 last 16.77) - open/close 13694 (time 1782615723.23 total 993.61 last 24.46) - open/close 13919 (time 1782615733.85 total 1004.23 last 21.18) - open/close 14087 (time 1782615744.18 total 1014.56 last 16.26) - open/close 14285 (time 1782615755.14 total 1025.52 last 18.08) - open/close 14442 (time 1782615765.69 total 1036.07 last 14.88) - open/close 14698 (time 1782615775.70 total 1046.08 last 25.57) - open/close 14961 (time 1782615785.70 total 1056.08 last 26.28) - open/close 15155 (time 1782615795.71 total 1066.09 last 19.39) - open/close 15282 (time 1782615806.64 total 1077.02 last 11.62) - open/close 15563 (time 1782615817.47 total 1087.85 last 25.94) - open/close 15962 (time 1782615827.48 total 1097.86 last 39.89) - open/close 16890 (time 1782615837.49 total 1107.87 last 92.71) - open/close 17694 (time 1782615847.49 total 1117.87 last 80.39) - open/close 18526 (time 1782615857.49 total 1127.87 last 83.15) - open/close 19334 (time 1782615867.49 total 1137.87 last 80.79) - open/close 20000 (time 1782615874.69 total 1145.07 last 92.53) - open/close 20910 (time 1782615884.71 total 1155.09 last 90.84) - open/close 21625 (time 1782615894.72 total 1165.10 last 71.43) - open/close 22345 (time 1782615904.73 total 1175.11 last 71.92) - open/close 23134 (time 1782615914.74 total 1185.12 last 78.86) - open/close 23875 (time 1782615924.74 total 1195.12 last 74.07) - open/close 24744 (time 1782615934.75 total 1205.13 last 86.82) - open/close 25499 (time 1782615944.76 total 1215.14 last 75.40) - open/close 26204 (time 1782615954.77 total 1225.15 last 70.43) - open/close 27101 (time 1782615964.77 total 1235.15 last 89.69) - open/close 28007 (time 1782615974.78 total 1245.16 last 90.53) - open/close 28891 (time 1782615984.78 total 1255.16 last 88.38) - open/close 29687 (time 1782615994.81 total 1265.19 last 79.42) - open/close 30000 (time 1782615999.14 total 1269.52 last 72.17) - open/close 30760 (time 1782616009.16 total 1279.54 last 75.90) - open/close 31649 (time 1782616019.16 total 1289.54 last 88.90) - open/close 32559 (time 1782616029.17 total 1299.55 last 90.91) - open/close 33348 (time 1782616039.17 total 1309.55 last 78.89) - open/close 34289 (time 1782616049.18 total 1319.56 last 94.03) - open/close 35195 (time 1782616059.18 total 1329.56 last 90.58) - open/close 36080 (time 1782616069.21 total 1339.59 last 88.22) - open/close 36796 (time 1782616079.22 total 1349.60 last 71.52) - open/close 37485 (time 1782616089.22 total 1359.60 last 68.89) - open/close 38212 (time 1782616099.23 total 1369.61 last 72.65) - open/close 39002 (time 1782616109.23 total 1379.61 last 78.96) - open/close 39806 (time 1782616119.25 total 1389.63 last 80.31) - open/close 40000 (time 1782616122.12 total 1392.50 last 67.53) - open/close 40798 (time 1782616132.12 total 1402.50 last 79.77) - open/close 41758 (time 1782616142.13 total 1412.52 last 95.88) - open/close 42547 (time 1782616152.16 total 1422.54 last 78.72) - open/close 43505 (time 1782616162.17 total 1432.55 last 95.72) - open/close 44341 (time 1782616172.18 total 1442.56 last 83.51) - open/close 45139 (time 1782616182.19 total 1452.57 last 79.72) - open/close 46009 (time 1782616192.20 total 1462.58 last 86.90) - open/close 46935 (time 1782616202.22 total 1472.60 last 92.43) - open/close 47684 (time 1782616212.23 total 1482.61 last 74.84) - open/close 48650 (time 1782616222.24 total 1492.62 last 96.50) - open/close 49535 (time 1782616232.25 total 1502.63 last 88.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1503.71 seconds: 33.01 ops/second - unlinked 0 (time 1782616238 ; total 0 ; last 0) - unlinked 10000 (time 1782616364 ; total 126 ; last 126) - unlinked 20000 (time 1782616483 ; total 245 ; last 119) - unlinked 30000 (time 1782616603 ; total 365 ; last 120) - unlinked 40000 (time 1782616729 ; total 491 ; last 126) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 608 seconds: 81.629936 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg149-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg149-server unloading modules via unload_modules_local on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing unload_modules_local oleg149-server: modules unloaded. pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2