== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:00:47 (1787738447) start mds service on oleg210-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg210-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 170 (time 1787738560.80 total 11.53 last 14.74) - open/close 331 (time 1787738570.80 total 21.54 last 16.10) - open/close 461 (time 1787738580.81 total 31.54 last 12.99) - open/close 537 (time 1787738590.94 total 41.67 last 7.50) - open/close 746 (time 1787738601.91 total 52.64 last 19.04) - open/close 986 (time 1787738612.46 total 63.19 last 22.76) - open/close 1125 (time 1787738622.65 total 73.39 last 13.64) - open/close 1333 (time 1787738633.12 total 83.85 last 19.88) - open/close 1503 (time 1787738644.66 total 95.39 last 14.73) - open/close 1576 (time 1787738654.87 total 105.61 last 7.15) - open/close 1843 (time 1787738664.88 total 115.61 last 26.69) - open/close 2006 (time 1787738674.90 total 125.63 last 16.27) - open/close 2122 (time 1787738685.22 total 135.95 last 11.24) - open/close 2333 (time 1787738695.33 total 146.06 last 20.86) - open/close 2459 (time 1787738705.57 total 156.30 last 12.30) - open/close 2615 (time 1787738718.64 total 169.37 last 11.94) - open/close 2772 (time 1787738731.07 total 181.80 last 12.63) - open/close 2913 (time 1787738741.28 total 192.01 last 13.80) - open/close 3034 (time 1787738751.28 total 202.01 last 12.10) - open/close 3147 (time 1787738761.43 total 212.16 last 11.14) - open/close 3414 (time 1787738771.44 total 222.17 last 26.67) - open/close 3655 (time 1787738782.05 total 232.78 last 22.71) - open/close 3828 (time 1787738792.35 total 243.08 last 16.80) - open/close 4075 (time 1787738802.67 total 253.40 last 23.93) - open/close 4338 (time 1787738812.87 total 263.60 last 25.78) - open/close 4486 (time 1787738823.52 total 274.25 last 13.89) - open/close 4636 (time 1787738833.95 total 284.68 last 14.38) - open/close 4727 (time 1787738845.41 total 296.14 last 7.94) - open/close 4989 (time 1787738856.16 total 306.89 last 24.37) - open/close 5336 (time 1787738866.42 total 317.15 last 33.81) - open/close 5624 (time 1787738876.43 total 327.16 last 28.78) - open/close 5932 (time 1787738886.45 total 337.18 last 30.74) - open/close 6115 (time 1787738897.37 total 348.10 last 16.75) - open/close 6304 (time 1787738907.37 total 358.10 last 18.90) - open/close 6503 (time 1787738918.04 total 368.77 last 18.65) - open/close 6747 (time 1787738928.30 total 379.03 last 23.78) - open/close 6923 (time 1787738938.53 total 389.26 last 17.20) - open/close 7226 (time 1787738948.91 total 399.64 last 29.18) - open/close 7502 (time 1787738958.94 total 409.67 last 27.52) - open/close 7718 (time 1787738968.95 total 419.68 last 21.58) - open/close 7961 (time 1787738978.96 total 429.69 last 24.27) - open/close 8108 (time 1787738988.96 total 439.70 last 14.70) - open/close 8269 (time 1787738999.65 total 450.39 last 15.06) - open/close 8443 (time 1787739009.66 total 460.39 last 17.39) - open/close 8560 (time 1787739019.81 total 470.54 last 11.53) - open/close 8820 (time 1787739029.82 total 480.55 last 25.97) - open/close 9016 (time 1787739040.03 total 490.76 last 19.20) - open/close 9152 (time 1787739051.05 total 501.78 last 12.35) - open/close 9359 (time 1787739061.08 total 511.81 last 20.63) - open/close 9546 (time 1787739071.84 total 522.57 last 17.38) - open/close 9677 (time 1787739082.84 total 533.57 last 11.91) - open/close 9754 (time 1787739093.81 total 544.54 last 7.02) - open/close 9978 (time 1787739103.82 total 554.55 last 22.37) - open/close 10192 (time 1787739113.83 total 564.56 last 21.38) - open/close 10282 (time 1787739124.47 total 575.20 last 8.46) - open/close 10466 (time 1787739135.21 total 585.94 last 17.13) - open/close 10572 (time 1787739145.36 total 596.10 last 10.44) - open/close 10676 (time 1787739156.60 total 607.33 last 9.25) - open/close 10829 (time 1787739167.46 total 618.19 last 14.09) - open/close 11121 (time 1787739178.32 total 629.05 last 26.88) - open/close 11227 (time 1787739188.35 total 639.08 last 10.57) - open/close 11443 (time 1787739198.87 total 649.60 last 20.53) - open/close 11636 (time 1787739210.65 total 661.38 last 16.38) - open/close 11855 (time 1787739220.67 total 671.40 last 21.86) - open/close 12054 (time 1787739231.16 total 681.89 last 18.96) - open/close 12272 (time 1787739241.24 total 691.97 last 21.64) - open/close 12317 (time 1787739251.63 total 702.36 last 4.33) - open/close 12431 (time 1787739264.48 total 715.21 last 8.87) - open/close 12590 (time 1787739274.50 total 725.23 last 15.87) - open/close 12729 (time 1787739284.53 total 735.26 last 13.86) - open/close 13010 (time 1787739294.53 total 745.26 last 28.09) - open/close 13213 (time 1787739304.53 total 755.27 last 20.29) - open/close 13449 (time 1787739314.54 total 765.28 last 23.57) - open/close 13745 (time 1787739324.55 total 775.28 last 29.58) - open/close 13964 (time 1787739334.57 total 785.30 last 21.86) - open/close 14246 (time 1787739344.58 total 795.31 last 28.18) - open/close 14399 (time 1787739354.74 total 805.47 last 15.06) - open/close 14467 (time 1787739364.74 total 815.47 last 6.80) - open/close 14708 (time 1787739375.35 total 826.08 last 22.72) - open/close 14964 (time 1787739386.02 total 836.75 last 23.99) - open/close 15238 (time 1787739396.03 total 846.76 last 27.38) - open/close 15497 (time 1787739411.83 total 862.57 last 16.39) - open/close 15563 (time 1787739422.61 total 873.35 last 6.12) - open/close 15627 (time 1787739433.70 total 884.43 last 5.77) - open/close 15735 (time 1787739443.70 total 894.44 last 10.80) - open/close 15894 (time 1787739454.57 total 905.30 last 14.64) - open/close 16137 (time 1787739464.80 total 915.54 last 23.73) - open/close 16288 (time 1787739475.24 total 925.97 last 14.47) - open/close 16464 (time 1787739485.27 total 936.00 last 17.55) - open/close 16564 (time 1787739495.70 total 946.43 last 9.59) - open/close 16696 (time 1787739506.60 total 957.33 last 12.11) - open/close 16911 (time 1787739516.61 total 967.34 last 21.49) - open/close 17065 (time 1787739526.62 total 977.35 last 15.38) - open/close 17276 (time 1787739537.07 total 987.80 last 20.20) - open/close 17473 (time 1787739547.22 total 997.95 last 19.40) - open/close 17623 (time 1787739557.29 total 1008.02 last 14.90) - open/close 17827 (time 1787739567.29 total 1018.02 last 20.40) - open/close 17939 (time 1787739577.33 total 1028.07 last 11.15) - open/close 18115 (time 1787739587.35 total 1038.08 last 17.58) - open/close 18221 (time 1787739598.07 total 1048.80 last 9.89) - open/close 18282 (time 1787739608.10 total 1058.84 last 6.08) - open/close 19077 (time 1787739618.11 total 1068.84 last 79.45) - open/close 20000 (time 1787739627.14 total 1077.88 last 102.17) - open/close 20884 (time 1787739637.15 total 1087.88 last 88.36) - open/close 21722 (time 1787739647.15 total 1097.89 last 83.76) - open/close 22685 (time 1787739657.16 total 1107.89 last 96.24) - open/close 23679 (time 1787739667.17 total 1117.90 last 99.35) - open/close 24574 (time 1787739677.17 total 1127.91 last 89.42) - open/close 25421 (time 1787739687.18 total 1137.91 last 84.67) - open/close 26370 (time 1787739697.18 total 1147.92 last 94.85) - open/close 27303 (time 1787739707.19 total 1157.92 last 93.28) - open/close 28117 (time 1787739717.19 total 1167.92 last 81.37) - open/close 28919 (time 1787739727.19 total 1177.93 last 80.17) - open/close 29821 (time 1787739737.20 total 1187.93 last 90.18) - open/close 30000 (time 1787739739.21 total 1189.94 last 89.02) - open/close 30753 (time 1787739749.21 total 1199.94 last 75.29) - open/close 31461 (time 1787739759.22 total 1209.95 last 70.73) - open/close 32095 (time 1787739769.23 total 1219.97 last 63.29) - open/close 32804 (time 1787739779.27 total 1230.00 last 70.68) - open/close 33515 (time 1787739789.27 total 1240.00 last 71.08) - open/close 34351 (time 1787739799.28 total 1250.01 last 83.49) - open/close 35079 (time 1787739809.28 total 1260.02 last 72.78) - open/close 36164 (time 1787739819.29 total 1270.03 last 108.38) - open/close 37152 (time 1787739829.30 total 1280.03 last 98.77) - open/close 37906 (time 1787739839.30 total 1290.03 last 75.36) - open/close 38614 (time 1787739849.31 total 1300.04 last 70.74) - open/close 39286 (time 1787739859.32 total 1310.06 last 67.11) - open/close 40000 (time 1787739868.21 total 1318.94 last 80.38) - open/close 40881 (time 1787739878.21 total 1328.94 last 88.05) - open/close 41714 (time 1787739888.21 total 1338.95 last 83.27) - open/close 42512 (time 1787739898.22 total 1348.95 last 79.77) - open/close 43376 (time 1787739908.23 total 1358.96 last 86.29) - open/close 44325 (time 1787739918.23 total 1368.97 last 94.86) - open/close 45339 (time 1787739928.24 total 1378.97 last 101.36) - open/close 46132 (time 1787739938.24 total 1388.97 last 79.29) - open/close 47052 (time 1787739948.24 total 1398.97 last 92.00) - open/close 47907 (time 1787739958.25 total 1408.98 last 85.42) - open/close 48653 (time 1787739968.26 total 1418.99 last 74.51) - open/close 49325 (time 1787739978.27 total 1429.01 last 67.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1433.41 seconds: 34.62 ops/second - unlinked 0 (time 1787739987 ; total 0 ; last 0) - unlinked 10000 (time 1787740114 ; total 127 ; last 127) - unlinked 20000 (time 1787740231 ; total 244 ; last 117) - unlinked 30000 (time 1787740358 ; total 371 ; last 127) - unlinked 40000 (time 1787740468 ; total 481 ; last 110) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 557 seconds: 89.104126 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded. pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2