== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:10:24 (1777579824) start mds service on oleg455-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg455-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg455-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1777579929.62 total 12.32 last 18.75) - open/close 443 (time 1777579940.16 total 22.86 last 20.11) - open/close 724 (time 1777579951.10 total 33.80 last 25.70) - open/close 992 (time 1777579961.74 total 44.44 last 25.17) - open/close 1235 (time 1777579971.76 total 54.46 last 24.27) - open/close 1446 (time 1777579981.76 total 64.46 last 21.09) - open/close 1665 (time 1777579992.70 total 75.40 last 20.02) - open/close 1848 (time 1777580003.15 total 85.85 last 17.51) - open/close 2064 (time 1777580013.17 total 95.87 last 21.55) - open/close 2269 (time 1777580023.35 total 106.05 last 20.14) - open/close 2520 (time 1777580033.35 total 116.06 last 25.09) - open/close 2745 (time 1777580044.06 total 126.76 last 21.03) - open/close 2831 (time 1777580054.97 total 137.67 last 7.88) - open/close 2965 (time 1777580064.98 total 147.68 last 13.39) - open/close 3076 (time 1777580074.99 total 157.69 last 11.09) - open/close 3229 (time 1777580084.99 total 167.69 last 15.29) - open/close 3454 (time 1777580095.00 total 177.70 last 22.48) - open/close 3777 (time 1777580105.61 total 188.31 last 30.46) - open/close 3933 (time 1777580115.79 total 198.49 last 15.32) - open/close 4055 (time 1777580126.30 total 209.01 last 11.61) - open/close 4302 (time 1777580137.53 total 220.23 last 22.01) - open/close 4502 (time 1777580149.51 total 232.21 last 16.69) - open/close 4644 (time 1777580159.53 total 242.23 last 14.17) - open/close 4779 (time 1777580169.53 total 252.23 last 13.50) - open/close 4967 (time 1777580179.53 total 262.23 last 18.80) - open/close 5201 (time 1777580190.01 total 272.72 last 22.32) - open/close 5479 (time 1777580200.31 total 283.02 last 26.99) - open/close 5713 (time 1777580210.32 total 293.02 last 23.39) - open/close 5916 (time 1777580220.33 total 303.03 last 20.28) - open/close 6022 (time 1777580231.10 total 313.80 last 9.84) - open/close 6167 (time 1777580241.11 total 323.81 last 14.48) - open/close 6349 (time 1777580251.12 total 333.82 last 18.17) - open/close 6527 (time 1777580261.13 total 343.84 last 17.78) - open/close 6714 (time 1777580271.44 total 354.14 last 18.14) - open/close 7003 (time 1777580281.78 total 364.48 last 27.95) - open/close 7221 (time 1777580292.68 total 375.38 last 20.01) - open/close 7340 (time 1777580302.97 total 385.67 last 11.56) - open/close 7427 (time 1777580312.99 total 395.70 last 8.68) - open/close 7688 (time 1777580324.06 total 406.76 last 23.59) - open/close 7912 (time 1777580334.07 total 416.77 last 22.36) - open/close 8100 (time 1777580344.12 total 426.82 last 18.72) - open/close 8292 (time 1777580355.00 total 437.71 last 17.63) - open/close 8473 (time 1777580365.02 total 447.72 last 18.07) - open/close 8598 (time 1777580375.04 total 457.74 last 12.47) - open/close 8814 (time 1777580385.06 total 467.76 last 21.56) - open/close 9043 (time 1777580395.40 total 478.10 last 22.15) - open/close 9265 (time 1777580406.61 total 489.31 last 19.80) - open/close 9433 (time 1777580416.83 total 499.53 last 16.44) - open/close 9559 (time 1777580427.72 total 510.42 last 11.57) - open/close 9708 (time 1777580437.91 total 520.61 last 14.61) - open/close 9855 (time 1777580447.93 total 530.63 last 14.67) - open/close 10000 (time 1777580455.91 total 538.61 last 18.18) - open/close 10262 (time 1777580465.91 total 548.61 last 26.18) - open/close 10479 (time 1777580475.94 total 558.64 last 21.65) - open/close 10771 (time 1777580485.96 total 568.66 last 29.14) - open/close 10968 (time 1777580495.98 total 578.68 last 19.66) - open/close 11217 (time 1777580506.21 total 588.91 last 24.35) - open/close 11394 (time 1777580516.94 total 599.64 last 16.50) - open/close 11496 (time 1777580526.95 total 609.65 last 10.19) - open/close 11579 (time 1777580537.43 total 620.13 last 7.92) - open/close 11802 (time 1777580547.66 total 630.36 last 21.80) - open/close 12015 (time 1777580558.57 total 641.27 last 19.51) - open/close 12128 (time 1777580569.92 total 652.62 last 9.95) - open/close 12375 (time 1777580580.35 total 663.05 last 23.69) - open/close 12585 (time 1777580590.38 total 673.08 last 20.93) - open/close 12792 (time 1777580600.55 total 683.25 last 20.36) - open/close 13041 (time 1777580610.94 total 693.64 last 23.95) - open/close 13259 (time 1777580621.71 total 704.41 last 20.25) - open/close 13516 (time 1777580632.20 total 714.90 last 24.49) - open/close 13805 (time 1777580642.21 total 724.91 last 28.88) - open/close 14013 (time 1777580654.04 total 736.74 last 17.58) - open/close 14205 (time 1777580664.82 total 747.52 last 17.81) - open/close 14429 (time 1777580674.83 total 757.53 last 22.36) - open/close 14670 (time 1777580687.50 total 770.20 last 19.03) - open/close 14865 (time 1777580698.57 total 781.27 last 17.61) - open/close 15129 (time 1777580708.58 total 791.28 last 26.38) - open/close 15394 (time 1777580719.35 total 802.05 last 24.59) - open/close 15645 (time 1777580729.68 total 812.39 last 24.30) - open/close 15915 (time 1777580739.69 total 822.39 last 26.99) - open/close 16070 (time 1777580750.05 total 832.75 last 14.96) - open/close 16221 (time 1777580760.28 total 842.98 last 14.76) - open/close 16455 (time 1777580770.81 total 853.51 last 22.22) - open/close 16703 (time 1777580781.19 total 863.89 last 23.90) - open/close 16824 (time 1777580795.02 total 877.72 last 8.75) - open/close 17044 (time 1777580805.04 total 887.74 last 21.95) - open/close 17309 (time 1777580815.07 total 897.77 last 26.42) - open/close 17552 (time 1777580825.07 total 907.78 last 24.28) - open/close 17815 (time 1777580835.79 total 918.49 last 24.55) - open/close 18029 (time 1777580845.80 total 928.50 last 21.38) - open/close 18240 (time 1777580857.43 total 940.14 last 18.14) - open/close 18458 (time 1777580868.29 total 950.99 last 20.09) - open/close 18960 (time 1777580878.30 total 961.00 last 50.16) - open/close 19681 (time 1777580888.30 total 971.00 last 72.10) - open/close 20000 (time 1777580892.31 total 975.01 last 79.42) - open/close 20740 (time 1777580902.32 total 985.02 last 73.94) - open/close 21408 (time 1777580912.33 total 995.03 last 66.74) - open/close 22159 (time 1777580922.33 total 1005.03 last 75.10) - open/close 22921 (time 1777580932.34 total 1015.04 last 76.14) - open/close 23797 (time 1777580942.34 total 1025.04 last 87.58) - open/close 24559 (time 1777580952.35 total 1035.05 last 76.15) - open/close 25362 (time 1777580962.35 total 1045.05 last 80.29) - open/close 26123 (time 1777580972.35 total 1055.05 last 76.06) - open/close 26839 (time 1777580982.36 total 1065.06 last 71.53) - open/close 27515 (time 1777580992.37 total 1075.07 last 67.53) - open/close 28094 (time 1777581002.39 total 1085.09 last 57.79) - open/close 28652 (time 1777581012.41 total 1095.11 last 55.70) - open/close 29220 (time 1777581022.42 total 1105.12 last 56.75) - open/close 29902 (time 1777581032.43 total 1115.13 last 68.15) - open/close 30476 (time 1777581042.46 total 1125.16 last 57.21) - open/close 31115 (time 1777581052.46 total 1135.16 last 63.89) - open/close 31890 (time 1777581062.47 total 1145.17 last 77.46) - open/close 32730 (time 1777581072.47 total 1155.17 last 83.97) - open/close 33427 (time 1777581082.48 total 1165.18 last 69.64) - open/close 34163 (time 1777581092.49 total 1175.19 last 73.53) - open/close 34886 (time 1777581102.49 total 1185.19 last 72.29) - open/close 35595 (time 1777581112.50 total 1195.20 last 70.82) - open/close 36244 (time 1777581122.51 total 1205.21 last 64.85) - open/close 37044 (time 1777581132.51 total 1215.21 last 79.99) - open/close 37816 (time 1777581142.53 total 1225.23 last 77.07) - open/close 38577 (time 1777581152.53 total 1235.24 last 76.04) - open/close 39322 (time 1777581162.55 total 1245.25 last 74.39) - open/close 40000 (time 1777581170.76 total 1253.46 last 82.57) - open/close 40712 (time 1777581180.77 total 1263.48 last 71.10) - open/close 41399 (time 1777581190.78 total 1273.48 last 68.64) - open/close 42169 (time 1777581200.80 total 1283.50 last 76.87) - open/close 42987 (time 1777581210.81 total 1293.51 last 81.75) - open/close 43694 (time 1777581220.81 total 1303.51 last 70.69) - open/close 44499 (time 1777581230.81 total 1313.51 last 80.45) - open/close 45259 (time 1777581240.83 total 1323.53 last 75.90) - open/close 46048 (time 1777581250.83 total 1333.53 last 78.87) - open/close 46783 (time 1777581260.84 total 1343.54 last 73.43) - open/close 47393 (time 1777581270.85 total 1353.55 last 60.95) - open/close 48240 (time 1777581280.86 total 1363.56 last 84.63) - open/close 48846 (time 1777581290.87 total 1373.58 last 60.49) - open/close 49587 (time 1777581300.89 total 1383.59 last 73.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1384.20 seconds: 35.86 ops/second - unlinked 0 (time 1777581306 ; total 0 ; last 0) - unlinked 10000 (time 1777581445 ; total 139 ; last 139) - unlinked 20000 (time 1777581578 ; total 272 ; last 133) - unlinked 30000 (time 1777581712 ; total 406 ; last 134) - unlinked 40000 (time 1777581848 ; total 542 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 667 seconds: 74.410797 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2