== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:53:57 (1787846037) start mds service on oleg209-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg209-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 66 (time 1787846154.82 total 10.01 last 6.59) - open/close 191 (time 1787846165.47 total 20.67 last 11.73) - open/close 451 (time 1787846175.49 total 30.68 last 25.95) - open/close 653 (time 1787846185.50 total 40.69 last 20.18) - open/close 704 (time 1787846196.13 total 51.32 last 4.80) - open/close 834 (time 1787846206.14 total 61.33 last 12.99) - open/close 1139 (time 1787846217.10 total 72.30 last 27.82) - open/close 1270 (time 1787846227.68 total 82.88 last 12.38) - open/close 1474 (time 1787846238.29 total 93.48 last 19.24) - open/close 1595 (time 1787846249.23 total 104.42 last 11.06) - open/close 1729 (time 1787846259.25 total 114.44 last 13.38) - open/close 1855 (time 1787846269.89 total 125.09 last 11.83) - open/close 1965 (time 1787846280.80 total 135.99 last 10.09) - open/close 2076 (time 1787846290.81 total 146.01 last 11.08) - open/close 2323 (time 1787846301.07 total 156.26 last 24.08) - open/close 2420 (time 1787846311.66 total 166.85 last 9.16) - open/close 2565 (time 1787846322.42 total 177.61 last 13.48) - open/close 2746 (time 1787846333.31 total 188.51 last 16.61) - open/close 2872 (time 1787846343.62 total 198.82 last 12.22) - open/close 2974 (time 1787846353.63 total 208.82 last 10.19) - open/close 3105 (time 1787846364.10 total 219.30 last 12.51) - open/close 3192 (time 1787846374.41 total 229.61 last 8.44) - open/close 3410 (time 1787846385.28 total 240.47 last 20.06) - open/close 3557 (time 1787846395.35 total 250.54 last 14.60) - open/close 3790 (time 1787846406.92 total 262.11 last 20.14) - open/close 3917 (time 1787846418.12 total 273.31 last 11.34) - open/close 4019 (time 1787846430.93 total 286.12 last 7.96) - open/close 4113 (time 1787846443.90 total 299.09 last 7.25) - open/close 4239 (time 1787846453.96 total 309.16 last 12.52) - open/close 4305 (time 1787846464.80 total 319.99 last 6.09) - open/close 4476 (time 1787846474.80 total 329.99 last 17.10) - open/close 4703 (time 1787846485.20 total 340.39 last 21.82) - open/close 4873 (time 1787846495.24 total 350.43 last 16.93) - open/close 5009 (time 1787846505.30 total 360.49 last 13.52) - open/close 5222 (time 1787846515.97 total 371.16 last 19.96) - open/close 5383 (time 1787846526.60 total 381.80 last 15.14) - open/close 5472 (time 1787846536.61 total 391.80 last 8.90) - open/close 5705 (time 1787846547.60 total 402.79 last 21.20) - open/close 5956 (time 1787846557.62 total 412.81 last 25.04) - open/close 6152 (time 1787846567.62 total 422.82 last 19.59) - open/close 6312 (time 1787846579.80 total 434.99 last 13.14) - open/close 6563 (time 1787846591.35 total 446.54 last 21.73) - open/close 6781 (time 1787846602.00 total 457.20 last 20.46) - open/close 6975 (time 1787846612.51 total 467.70 last 18.47) - open/close 7198 (time 1787846622.89 total 478.08 last 21.48) - open/close 7437 (time 1787846633.82 total 489.01 last 21.86) - open/close 7578 (time 1787846649.68 total 504.87 last 8.89) - open/close 7831 (time 1787846659.69 total 514.88 last 25.27) - open/close 8064 (time 1787846670.73 total 525.92 last 21.12) - open/close 8241 (time 1787846681.48 total 536.67 last 16.46) - open/close 8531 (time 1787846691.50 total 546.69 last 28.94) - open/close 8696 (time 1787846702.34 total 557.54 last 15.21) - open/close 8852 (time 1787846714.89 total 570.08 last 12.44) - open/close 9049 (time 1787846726.10 total 581.29 last 17.57) - open/close 9242 (time 1787846736.11 total 591.31 last 19.27) - open/close 9350 (time 1787846746.98 total 602.18 last 9.94) - open/close 9503 (time 1787846757.39 total 612.59 last 14.70) - open/close 9683 (time 1787846767.41 total 622.60 last 17.97) - open/close 9808 (time 1787846777.82 total 633.02 last 12.00) - open/close 10000 (time 1787846787.00 total 642.20 last 20.92) - open/close 10169 (time 1787846797.65 total 652.84 last 15.87) - open/close 10308 (time 1787846808.02 total 663.21 last 13.41) - open/close 10397 (time 1787846818.89 total 674.08 last 8.19) - open/close 10586 (time 1787846828.91 total 684.10 last 18.87) - open/close 10729 (time 1787846839.45 total 694.64 last 13.57) - open/close 10865 (time 1787846849.87 total 705.07 last 13.04) - open/close 10979 (time 1787846860.12 total 715.31 last 11.13) - open/close 11155 (time 1787846870.14 total 725.33 last 17.56) - open/close 11326 (time 1787846880.98 total 736.17 last 15.78) - open/close 11461 (time 1787846891.04 total 746.23 last 13.42) - open/close 11660 (time 1787846901.17 total 756.37 last 19.64) - open/close 11760 (time 1787846911.94 total 767.14 last 9.28) - open/close 11954 (time 1787846921.96 total 777.16 last 19.36) - open/close 12146 (time 1787846932.07 total 787.26 last 19.00) - open/close 12259 (time 1787846942.47 total 797.67 last 10.86) - open/close 12398 (time 1787846953.57 total 808.77 last 12.52) - open/close 12531 (time 1787846964.57 total 819.76 last 12.10) - open/close 12703 (time 1787846974.61 total 829.80 last 17.13) - open/close 12866 (time 1787846985.58 total 840.77 last 14.86) - open/close 12995 (time 1787846995.92 total 851.11 last 12.47) - open/close 13109 (time 1787847006.60 total 861.80 last 10.67) - open/close 13321 (time 1787847017.74 total 872.93 last 19.03) - open/close 13521 (time 1787847027.78 total 882.98 last 19.91) - open/close 13707 (time 1787847037.79 total 892.99 last 18.58) - open/close 13882 (time 1787847047.80 total 902.99 last 17.49) - open/close 14030 (time 1787847058.63 total 913.83 last 13.66) - open/close 14240 (time 1787847068.64 total 923.84 last 20.98) - open/close 14369 (time 1787847078.86 total 934.05 last 12.63) - open/close 14502 (time 1787847088.86 total 944.06 last 13.29) - open/close 14602 (time 1787847098.89 total 954.08 last 9.98) - open/close 14749 (time 1787847109.33 total 964.52 last 14.08) - open/close 15003 (time 1787847119.34 total 974.54 last 25.37) - open/close 15106 (time 1787847129.93 total 985.12 last 9.73) - open/close 15337 (time 1787847140.56 total 995.75 last 21.74) - open/close 15633 (time 1787847151.25 total 1006.45 last 27.67) - open/close 15845 (time 1787847161.44 total 1016.64 last 20.80) - open/close 16002 (time 1787847171.48 total 1026.67 last 15.64) - open/close 16122 (time 1787847181.49 total 1036.68 last 11.99) - open/close 16304 (time 1787847192.39 total 1047.58 last 16.69) - open/close 16447 (time 1787847202.88 total 1058.07 last 13.63) - open/close 16651 (time 1787847213.20 total 1068.39 last 19.78) - open/close 16914 (time 1787847223.20 total 1078.40 last 26.28) - open/close 17637 (time 1787847233.21 total 1088.41 last 72.21) - open/close 18452 (time 1787847243.22 total 1098.41 last 81.49) - open/close 19282 (time 1787847253.23 total 1108.43 last 82.85) - open/close 20000 (time 1787847262.83 total 1118.03 last 74.80) - open/close 20880 (time 1787847272.84 total 1128.03 last 87.95) - open/close 21778 (time 1787847282.84 total 1138.03 last 89.77) - open/close 22454 (time 1787847292.84 total 1148.04 last 67.57) - open/close 23131 (time 1787847302.86 total 1158.05 last 67.62) - open/close 23860 (time 1787847312.86 total 1168.05 last 72.89) - open/close 24692 (time 1787847322.86 total 1178.05 last 83.17) - open/close 25632 (time 1787847332.86 total 1188.06 last 93.98) - open/close 26385 (time 1787847342.88 total 1198.07 last 75.18) - open/close 27274 (time 1787847352.89 total 1208.08 last 88.81) - open/close 28022 (time 1787847362.90 total 1218.09 last 74.74) - open/close 28761 (time 1787847372.90 total 1228.10 last 73.85) - open/close 29530 (time 1787847382.91 total 1238.10 last 76.87) - open/close 30000 (time 1787847388.17 total 1243.36 last 89.29) - open/close 30702 (time 1787847398.18 total 1253.37 last 70.16) - open/close 31487 (time 1787847408.19 total 1263.38 last 78.39) - open/close 32375 (time 1787847418.20 total 1273.39 last 88.74) - open/close 33368 (time 1787847428.20 total 1283.39 last 99.28) - open/close 34202 (time 1787847438.20 total 1293.40 last 83.35) - open/close 35063 (time 1787847448.20 total 1303.40 last 86.10) - open/close 35753 (time 1787847458.21 total 1313.41 last 68.93) - open/close 36513 (time 1787847468.21 total 1323.41 last 75.99) - open/close 37373 (time 1787847478.22 total 1333.41 last 85.98) - open/close 38167 (time 1787847488.22 total 1343.42 last 79.36) - open/close 38960 (time 1787847498.23 total 1353.42 last 79.27) - open/close 39723 (time 1787847508.23 total 1363.43 last 76.26) - open/close 40000 (time 1787847512.74 total 1367.93 last 61.49) - open/close 40764 (time 1787847522.74 total 1377.93 last 76.37) - open/close 41566 (time 1787847532.75 total 1387.95 last 80.10) - open/close 42434 (time 1787847542.76 total 1397.96 last 86.70) - open/close 43165 (time 1787847552.77 total 1407.97 last 73.04) - open/close 43887 (time 1787847562.78 total 1417.98 last 72.12) - open/close 44717 (time 1787847572.79 total 1427.98 last 82.96) - open/close 45445 (time 1787847582.80 total 1437.99 last 72.72) - open/close 46150 (time 1787847592.81 total 1448.00 last 70.43) - open/close 46879 (time 1787847602.83 total 1458.02 last 72.75) - open/close 47493 (time 1787847612.83 total 1468.03 last 61.36) - open/close 48078 (time 1787847622.84 total 1478.04 last 58.44) - open/close 48800 (time 1787847632.85 total 1488.04 last 72.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1497.49 seconds: 33.14 ops/second - unlinked 0 (time 1787847647 ; total 0 ; last 0) - unlinked 10000 (time 1787847780 ; total 133 ; last 133) - unlinked 20000 (time 1787847919 ; total 272 ; last 139) - unlinked 30000 (time 1787848076 ; total 429 ; last 157) - unlinked 40000 (time 1787848219 ; total 572 ; last 143) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 701 seconds: 70.800285 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: modules unloaded. pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2