== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:53:35 (1787122415) start mds service on oleg322-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg322-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1787122524.63 total 10.96 last 14.32) - open/close 320 (time 1787122534.85 total 21.18 last 15.96) - open/close 549 (time 1787122545.35 total 31.68 last 21.81) - open/close 807 (time 1787122555.36 total 41.69 last 25.76) - open/close 984 (time 1787122566.15 total 52.48 last 16.41) - open/close 1183 (time 1787122576.15 total 62.48 last 19.90) - open/close 1405 (time 1787122586.16 total 72.49 last 22.18) - open/close 1638 (time 1787122596.37 total 82.70 last 22.83) - open/close 1850 (time 1787122606.37 total 92.70 last 21.18) - open/close 2017 (time 1787122616.65 total 102.98 last 16.24) - open/close 2283 (time 1787122627.35 total 113.68 last 24.87) - open/close 2557 (time 1787122638.09 total 124.42 last 25.50) - open/close 2827 (time 1787122648.90 total 135.23 last 24.99) - open/close 3116 (time 1787122658.90 total 145.23 last 28.89) - open/close 3297 (time 1787122668.91 total 155.24 last 18.08) - open/close 3421 (time 1787122678.91 total 165.24 last 12.40) - open/close 3604 (time 1787122689.48 total 175.81 last 17.31) - open/close 3870 (time 1787122699.50 total 185.83 last 26.56) - open/close 4130 (time 1787122709.50 total 195.83 last 25.99) - open/close 4369 (time 1787122719.95 total 206.28 last 22.87) - open/close 4616 (time 1787122730.57 total 216.90 last 23.27) - open/close 4897 (time 1787122740.58 total 226.91 last 28.08) - open/close 5118 (time 1787122750.59 total 236.92 last 22.07) - open/close 5350 (time 1787122760.91 total 247.24 last 22.48) - open/close 5454 (time 1787122770.91 total 257.24 last 10.40) - open/close 5564 (time 1787122780.92 total 267.25 last 10.99) - open/close 5774 (time 1787122790.92 total 277.25 last 20.99) - open/close 5968 (time 1787122803.50 total 289.83 last 15.43) - open/close 6190 (time 1787122816.02 total 302.35 last 17.73) - open/close 6379 (time 1787122826.92 total 313.25 last 17.34) - open/close 6531 (time 1787122837.95 total 324.28 last 13.77) - open/close 6740 (time 1787122847.96 total 334.29 last 20.88) - open/close 6947 (time 1787122857.96 total 344.29 last 20.70) - open/close 7167 (time 1787122868.62 total 354.95 last 20.65) - open/close 7436 (time 1787122878.85 total 365.18 last 26.29) - open/close 7672 (time 1787122888.85 total 375.18 last 23.59) - open/close 7957 (time 1787122898.86 total 385.19 last 28.48) - open/close 8134 (time 1787122908.93 total 395.26 last 17.57) - open/close 8362 (time 1787122919.41 total 405.74 last 21.77) - open/close 8570 (time 1787122930.02 total 416.35 last 19.60) - open/close 8705 (time 1787122940.18 total 426.51 last 13.28) - open/close 8771 (time 1787122950.23 total 436.56 last 6.57) - open/close 8916 (time 1787122960.24 total 446.57 last 14.48) - open/close 9150 (time 1787122970.25 total 456.58 last 23.38) - open/close 9338 (time 1787122981.19 total 467.52 last 17.18) - open/close 9566 (time 1787122991.69 total 478.02 last 21.72) - open/close 9763 (time 1787123001.81 total 488.14 last 19.46) - open/close 10000 (time 1787123011.04 total 497.37 last 25.68) - open/close 10218 (time 1787123021.04 total 507.37 last 21.79) - open/close 10507 (time 1787123031.94 total 518.27 last 26.52) - open/close 10757 (time 1787123041.94 total 528.27 last 25.00) - open/close 10998 (time 1787123051.94 total 538.27 last 24.09) - open/close 11226 (time 1787123062.85 total 549.18 last 20.91) - open/close 11512 (time 1787123073.86 total 560.19 last 25.97) - open/close 11740 (time 1787123083.87 total 570.20 last 22.78) - open/close 11993 (time 1787123094.40 total 580.73 last 24.02) - open/close 12245 (time 1787123104.73 total 591.06 last 24.40) - open/close 12523 (time 1787123115.41 total 601.74 last 26.03) - open/close 12744 (time 1787123126.03 total 612.36 last 20.81) - open/close 12994 (time 1787123136.52 total 622.85 last 23.81) - open/close 13229 (time 1787123146.96 total 633.29 last 22.52) - open/close 13483 (time 1787123156.96 total 643.29 last 25.39) - open/close 13628 (time 1787123167.76 total 654.08 last 13.44) - open/close 13854 (time 1787123178.00 total 664.33 last 22.05) - open/close 14127 (time 1787123189.00 total 675.33 last 24.82) - open/close 14289 (time 1787123199.01 total 685.34 last 16.19) - open/close 14503 (time 1787123209.01 total 695.34 last 21.39) - open/close 14700 (time 1787123219.01 total 705.34 last 19.70) - open/close 14890 (time 1787123229.01 total 715.34 last 18.99) - open/close 15139 (time 1787123239.63 total 725.96 last 23.45) - open/close 15400 (time 1787123250.25 total 736.58 last 24.58) - open/close 15606 (time 1787123260.37 total 746.70 last 20.35) - open/close 15865 (time 1787123270.38 total 756.71 last 25.89) - open/close 16048 (time 1787123280.68 total 767.00 last 17.77) - open/close 16259 (time 1787123290.70 total 777.03 last 21.04) - open/close 16463 (time 1787123300.71 total 787.04 last 20.37) - open/close 16699 (time 1787123311.44 total 797.77 last 21.99) - open/close 16890 (time 1787123323.66 total 809.99 last 15.64) - open/close 17052 (time 1787123334.40 total 820.73 last 15.08) - open/close 17282 (time 1787123344.74 total 831.07 last 22.24) - open/close 17498 (time 1787123356.13 total 842.46 last 18.98) - open/close 17656 (time 1787123367.10 total 853.43 last 14.40) - open/close 17867 (time 1787123377.56 total 863.89 last 20.18) - open/close 18052 (time 1787123389.03 total 875.36 last 16.13) - open/close 18291 (time 1787123399.95 total 886.28 last 21.88) - open/close 18461 (time 1787123412.45 total 898.78 last 13.61) - open/close 18731 (time 1787123422.45 total 908.78 last 26.98) - open/close 18943 (time 1787123432.96 total 919.29 last 20.18) - open/close 19188 (time 1787123442.96 total 929.29 last 24.50) - open/close 19431 (time 1787123453.57 total 939.90 last 22.90) - open/close 19800 (time 1787123463.58 total 949.91 last 36.89) - open/close 20000 (time 1787123466.28 total 952.61 last 73.98) - open/close 20714 (time 1787123476.28 total 962.61 last 71.39) - open/close 21470 (time 1787123486.29 total 972.62 last 75.53) - open/close 22244 (time 1787123496.31 total 982.63 last 77.30) - open/close 23040 (time 1787123506.32 total 992.65 last 79.50) - open/close 23738 (time 1787123516.32 total 1002.65 last 69.80) - open/close 24442 (time 1787123526.32 total 1012.65 last 70.36) - open/close 25128 (time 1787123536.35 total 1022.68 last 68.42) - open/close 25828 (time 1787123546.36 total 1032.69 last 69.96) - open/close 26471 (time 1787123556.36 total 1042.69 last 64.25) - open/close 27065 (time 1787123566.37 total 1052.70 last 59.34) - open/close 27716 (time 1787123576.39 total 1062.72 last 65.00) - open/close 28354 (time 1787123586.40 total 1072.73 last 63.74) - open/close 28918 (time 1787123596.42 total 1082.75 last 56.29) - open/close 29495 (time 1787123606.42 total 1092.75 last 57.68) - open/close 29938 (time 1787123616.46 total 1102.79 last 44.13) - open/close 30509 (time 1787123626.47 total 1112.80 last 57.05) - open/close 30970 (time 1787123636.48 total 1122.81 last 46.05) - open/close 31631 (time 1787123646.48 total 1132.81 last 66.08) - open/close 32280 (time 1787123656.48 total 1142.81 last 64.89) - open/close 33077 (time 1787123666.49 total 1152.82 last 79.66) - open/close 33811 (time 1787123676.49 total 1162.82 last 73.37) - open/close 34490 (time 1787123686.50 total 1172.83 last 67.83) - open/close 35133 (time 1787123696.51 total 1182.84 last 64.26) - open/close 35804 (time 1787123706.51 total 1192.84 last 67.09) - open/close 36512 (time 1787123716.52 total 1202.85 last 70.74) - open/close 37199 (time 1787123726.52 total 1212.85 last 68.68) - open/close 37924 (time 1787123736.52 total 1222.85 last 72.49) - open/close 38592 (time 1787123746.53 total 1232.86 last 66.78) - open/close 39278 (time 1787123756.54 total 1242.87 last 68.49) - open/close 40000 (time 1787123766.45 total 1252.78 last 72.91) - open/close 40741 (time 1787123776.45 total 1262.78 last 74.09) - open/close 41459 (time 1787123786.45 total 1272.78 last 71.79) - open/close 42103 (time 1787123796.45 total 1282.78 last 64.39) - open/close 42866 (time 1787123806.46 total 1292.79 last 76.23) - open/close 43555 (time 1787123816.46 total 1302.79 last 68.89) - open/close 44275 (time 1787123826.47 total 1312.80 last 71.96) - open/close 45023 (time 1787123836.48 total 1322.81 last 74.69) - open/close 45747 (time 1787123846.49 total 1332.82 last 72.36) - open/close 46506 (time 1787123856.49 total 1342.82 last 75.88) - open/close 47218 (time 1787123866.50 total 1352.83 last 71.13) - open/close 47936 (time 1787123876.51 total 1362.84 last 71.73) - open/close 48705 (time 1787123886.51 total 1372.84 last 76.87) - open/close 49449 (time 1787123896.52 total 1382.85 last 74.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1385.80 seconds: 35.81 ops/second - unlinked 0 (time 1787123904 ; total 0 ; last 0) - unlinked 10000 (time 1787124058 ; total 154 ; last 154) - unlinked 20000 (time 1787124209 ; total 305 ; last 151) - unlinked 30000 (time 1787124365 ; total 461 ; last 156) - unlinked 40000 (time 1787124511 ; total 607 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 743 seconds: 66.798119 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: modules unloaded. pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2