== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:02:14 (1787580134) start mds service on oleg402-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg402-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 176 (time 1787580245.43 total 10.63 last 16.55) - open/close 377 (time 1787580255.56 total 20.76 last 19.84) - open/close 541 (time 1787580265.57 total 30.77 last 16.39) - open/close 743 (time 1787580275.57 total 40.77 last 20.19) - open/close 929 (time 1787580285.58 total 50.78 last 18.58) - open/close 1141 (time 1787580296.46 total 61.66 last 19.49) - open/close 1412 (time 1787580306.47 total 71.67 last 27.08) - open/close 1562 (time 1787580316.49 total 81.69 last 14.96) - open/close 1747 (time 1787580326.85 total 92.05 last 17.87) - open/close 1920 (time 1787580336.85 total 102.05 last 17.29) - open/close 2065 (time 1787580346.86 total 112.06 last 14.49) - open/close 2343 (time 1787580356.87 total 122.07 last 27.76) - open/close 2499 (time 1787580366.89 total 132.09 last 15.57) - open/close 2616 (time 1787580377.67 total 142.87 last 10.85) - open/close 2810 (time 1787580387.70 total 152.90 last 19.36) - open/close 2962 (time 1787580398.73 total 163.93 last 13.78) - open/close 3141 (time 1787580408.76 total 173.96 last 17.83) - open/close 3328 (time 1787580419.65 total 184.85 last 17.17) - open/close 3508 (time 1787580430.66 total 195.86 last 16.35) - open/close 3755 (time 1787580440.67 total 205.87 last 24.68) - open/close 3885 (time 1787580450.69 total 215.89 last 12.97) - open/close 4096 (time 1787580461.71 total 226.91 last 19.16) - open/close 4294 (time 1787580472.68 total 237.88 last 18.04) - open/close 4563 (time 1787580482.71 total 247.91 last 26.83) - open/close 4795 (time 1787580493.19 total 258.39 last 22.13) - open/close 5036 (time 1787580505.85 total 271.05 last 19.04) - open/close 5333 (time 1787580515.87 total 281.07 last 29.64) - open/close 5518 (time 1787580525.87 total 291.07 last 18.50) - open/close 5617 (time 1787580538.32 total 303.52 last 7.95) - open/close 5808 (time 1787580549.09 total 314.29 last 17.74) - open/close 6054 (time 1787580560.17 total 325.37 last 22.19) - open/close 6193 (time 1787580570.19 total 335.39 last 13.87) - open/close 6401 (time 1787580580.73 total 345.93 last 19.73) - open/close 6638 (time 1787580591.05 total 356.25 last 22.97) - open/close 6829 (time 1787580601.16 total 366.36 last 18.91) - open/close 7042 (time 1787580611.16 total 376.36 last 21.30) - open/close 7360 (time 1787580621.16 total 386.36 last 31.79) - open/close 7561 (time 1787580631.18 total 396.38 last 20.05) - open/close 7813 (time 1787580641.20 total 406.40 last 25.17) - open/close 8043 (time 1787580651.46 total 416.66 last 22.41) - open/close 8240 (time 1787580661.47 total 426.67 last 19.68) - open/close 8501 (time 1787580672.45 total 437.65 last 23.78) - open/close 8742 (time 1787580682.88 total 448.08 last 23.10) - open/close 8996 (time 1787580692.90 total 458.10 last 25.35) - open/close 9200 (time 1787580702.95 total 468.15 last 20.30) - open/close 9304 (time 1787580714.33 total 479.53 last 9.14) - open/close 9554 (time 1787580724.33 total 489.53 last 24.99) - open/close 9820 (time 1787580735.31 total 500.51 last 24.23) - open/close 10000 (time 1787580742.32 total 507.52 last 25.70) - open/close 10227 (time 1787580752.33 total 517.53 last 22.67) - open/close 10523 (time 1787580762.33 total 527.53 last 29.60) - open/close 10692 (time 1787580772.42 total 537.62 last 16.74) - open/close 10863 (time 1787580782.67 total 547.87 last 16.69) - open/close 11019 (time 1787580792.69 total 557.89 last 15.57) - open/close 11220 (time 1787580802.82 total 568.02 last 19.84) - open/close 11489 (time 1787580813.32 total 578.52 last 25.63) - open/close 11594 (time 1787580823.68 total 588.88 last 10.13) - open/close 11813 (time 1787580833.88 total 599.08 last 21.48) - open/close 12009 (time 1787580844.76 total 609.96 last 18.01) - open/close 12238 (time 1787580855.10 total 620.30 last 22.14) - open/close 12473 (time 1787580865.11 total 630.31 last 23.48) - open/close 12687 (time 1787580875.12 total 640.32 last 21.38) - open/close 12918 (time 1787580886.08 total 651.28 last 21.07) - open/close 13145 (time 1787580896.40 total 661.60 last 22.00) - open/close 13406 (time 1787580906.76 total 671.96 last 25.19) - open/close 13627 (time 1787580916.77 total 681.97 last 22.08) - open/close 13902 (time 1787580926.78 total 691.98 last 27.47) - open/close 14126 (time 1787580937.35 total 702.55 last 21.18) - open/close 14357 (time 1787580947.64 total 712.84 last 22.44) - open/close 14547 (time 1787580957.65 total 722.85 last 18.98) - open/close 14746 (time 1787580967.67 total 732.87 last 19.87) - open/close 14978 (time 1787580977.86 total 743.06 last 22.76) - open/close 15211 (time 1787580987.87 total 753.07 last 23.28) - open/close 15349 (time 1787580997.88 total 763.08 last 13.79) - open/close 15569 (time 1787581007.88 total 773.08 last 22.00) - open/close 15765 (time 1787581017.89 total 783.09 last 19.59) - open/close 15941 (time 1787581028.68 total 793.88 last 16.32) - open/close 16105 (time 1787581038.69 total 803.89 last 16.38) - open/close 16286 (time 1787581048.90 total 814.10 last 17.72) - open/close 16551 (time 1787581058.92 total 824.12 last 26.45) - open/close 16688 (time 1787581070.86 total 836.06 last 11.48) - open/close 16896 (time 1787581080.86 total 846.06 last 20.79) - open/close 17099 (time 1787581090.90 total 856.10 last 20.23) - open/close 17273 (time 1787581101.78 total 866.98 last 15.99) - open/close 17530 (time 1787581111.78 total 876.98 last 25.69) - open/close 17750 (time 1787581122.45 total 887.65 last 20.62) - open/close 18034 (time 1787581133.44 total 898.64 last 25.82) - open/close 18223 (time 1787581144.41 total 909.61 last 17.23) - open/close 18346 (time 1787581154.89 total 920.09 last 11.74) - open/close 18536 (time 1787581164.90 total 930.10 last 18.99) - open/close 18760 (time 1787581174.97 total 940.17 last 22.24) - open/close 19067 (time 1787581185.66 total 950.86 last 28.71) - open/close 19306 (time 1787581195.84 total 961.04 last 23.48) - open/close 19908 (time 1787581205.85 total 971.05 last 60.14) - open/close 20583 (time 1787581215.86 total 981.06 last 67.46) - open/close 21248 (time 1787581225.86 total 991.06 last 66.48) - open/close 21974 (time 1787581235.88 total 1001.08 last 72.42) - open/close 22613 (time 1787581245.88 total 1011.09 last 63.89) - open/close 23230 (time 1787581255.90 total 1021.10 last 61.61) - open/close 23970 (time 1787581265.92 total 1031.12 last 73.86) - open/close 24669 (time 1787581275.92 total 1041.12 last 69.88) - open/close 25365 (time 1787581285.92 total 1051.12 last 69.58) - open/close 26148 (time 1787581295.93 total 1061.13 last 78.28) - open/close 26874 (time 1787581305.93 total 1071.13 last 72.57) - open/close 27588 (time 1787581315.93 total 1081.13 last 71.38) - open/close 28179 (time 1787581325.96 total 1091.16 last 58.98) - open/close 28776 (time 1787581335.96 total 1101.16 last 59.69) - open/close 29287 (time 1787581345.96 total 1111.16 last 51.09) - open/close 29981 (time 1787581355.98 total 1121.18 last 69.23) - open/close 30595 (time 1787581366.00 total 1131.20 last 61.32) - open/close 31212 (time 1787581376.00 total 1141.20 last 61.67) - open/close 31841 (time 1787581386.00 total 1151.20 last 62.90) - open/close 32478 (time 1787581396.01 total 1161.21 last 63.67) - open/close 33162 (time 1787581406.02 total 1171.22 last 68.29) - open/close 33798 (time 1787581416.04 total 1181.24 last 63.49) - open/close 34467 (time 1787581426.06 total 1191.26 last 66.79) - open/close 35117 (time 1787581436.07 total 1201.27 last 64.92) - open/close 35827 (time 1787581446.09 total 1211.29 last 70.87) - open/close 36532 (time 1787581456.09 total 1221.29 last 70.44) - open/close 37201 (time 1787581466.10 total 1231.31 last 66.83) - open/close 37847 (time 1787581476.11 total 1241.31 last 64.57) - open/close 38527 (time 1787581486.11 total 1251.31 last 67.98) - open/close 39168 (time 1787581496.12 total 1261.32 last 64.05) - open/close 39705 (time 1787581506.12 total 1271.32 last 53.69) - open/close 40000 (time 1787581511.18 total 1276.38 last 58.33) - open/close 40572 (time 1787581521.19 total 1286.39 last 57.15) - open/close 41160 (time 1787581531.19 total 1296.39 last 58.77) - open/close 41829 (time 1787581541.19 total 1306.39 last 66.90) - open/close 42389 (time 1787581551.22 total 1316.42 last 55.86) - open/close 42952 (time 1787581561.23 total 1326.43 last 56.24) - open/close 43582 (time 1787581571.24 total 1336.44 last 62.95) - open/close 44246 (time 1787581581.25 total 1346.45 last 66.32) - open/close 44850 (time 1787581591.25 total 1356.45 last 60.38) - open/close 45515 (time 1787581601.27 total 1366.47 last 66.39) - open/close 46102 (time 1787581611.28 total 1376.48 last 58.63) - open/close 46739 (time 1787581621.28 total 1386.48 last 63.69) - open/close 47344 (time 1787581631.29 total 1396.49 last 60.46) - open/close 48026 (time 1787581641.29 total 1406.49 last 68.17) - open/close 48655 (time 1787581651.31 total 1416.51 last 62.81) - open/close 49371 (time 1787581661.33 total 1426.53 last 71.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1430.61 seconds: 34.69 ops/second - unlinked 0 (time 1787581671 ; total 1 ; last 1) - unlinked 10000 (time 1787581829 ; total 159 ; last 158) - unlinked 20000 (time 1787581979 ; total 309 ; last 150) - unlinked 30000 (time 1787582126 ; total 456 ; last 147) - unlinked 40000 (time 1787582270 ; total 600 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 753 seconds: 65.911026 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: modules unloaded. pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2