== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:10:39 (1777576239) start mds service on oleg356-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg356-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 86 (time 1777576350.08 total 10.49 last 8.20) - open/close 300 (time 1777576360.08 total 20.49 last 21.40) - open/close 458 (time 1777576370.29 total 30.70 last 15.47) - open/close 573 (time 1777576380.32 total 40.73 last 11.47) - open/close 726 (time 1777576390.59 total 51.00 last 14.90) - open/close 884 (time 1777576401.22 total 61.64 last 14.86) - open/close 1096 (time 1777576411.24 total 71.65 last 21.17) - open/close 1356 (time 1777576421.25 total 81.66 last 25.97) - open/close 1499 (time 1777576432.37 total 92.78 last 12.86) - open/close 1687 (time 1777576443.31 total 103.72 last 17.18) - open/close 1837 (time 1777576457.69 total 118.10 last 10.43) - open/close 1926 (time 1777576467.70 total 128.12 last 8.89) - open/close 2007 (time 1777576477.73 total 138.14 last 8.08) - open/close 2254 (time 1777576487.75 total 148.16 last 24.64) - open/close 2442 (time 1777576497.78 total 158.19 last 18.74) - open/close 2645 (time 1777576508.20 total 168.61 last 19.49) - open/close 2851 (time 1777576518.59 total 179.00 last 19.83) - open/close 3090 (time 1777576528.59 total 189.01 last 23.89) - open/close 3276 (time 1777576539.46 total 199.87 last 17.12) - open/close 3521 (time 1777576550.40 total 210.82 last 22.38) - open/close 3623 (time 1777576560.63 total 221.05 last 9.97) - open/close 3666 (time 1777576572.97 total 233.39 last 3.48) - open/close 3800 (time 1777576585.07 total 245.48 last 11.08) - open/close 3992 (time 1777576595.08 total 255.49 last 19.19) - open/close 4131 (time 1777576605.08 total 265.49 last 13.90) - open/close 4379 (time 1777576615.09 total 275.50 last 24.77) - open/close 4649 (time 1777576625.10 total 285.51 last 26.96) - open/close 4819 (time 1777576635.32 total 295.73 last 16.63) - open/close 4890 (time 1777576646.97 total 307.38 last 6.10) - open/close 5063 (time 1777576657.34 total 317.75 last 16.69) - open/close 5293 (time 1777576669.56 total 329.97 last 18.82) - open/close 5553 (time 1777576680.38 total 340.79 last 24.03) - open/close 5752 (time 1777576690.38 total 350.80 last 19.89) - open/close 5952 (time 1777576700.61 total 361.02 last 19.55) - open/close 6113 (time 1777576710.84 total 371.25 last 15.74) - open/close 6266 (time 1777576721.32 total 381.73 last 14.60) - open/close 6471 (time 1777576733.76 total 394.17 last 16.48) - open/close 6741 (time 1777576744.63 total 405.05 last 24.83) - open/close 6964 (time 1777576755.83 total 416.25 last 19.91) - open/close 7229 (time 1777576765.93 total 426.34 last 26.26) - open/close 7371 (time 1777576775.93 total 436.35 last 14.19) - open/close 7544 (time 1777576786.81 total 447.22 last 15.91) - open/close 7625 (time 1777576797.56 total 457.97 last 7.53) - open/close 7759 (time 1777576807.57 total 467.99 last 13.38) - open/close 7960 (time 1777576817.90 total 478.31 last 19.47) - open/close 8121 (time 1777576828.52 total 488.93 last 15.16) - open/close 8283 (time 1777576840.68 total 501.10 last 13.32) - open/close 8364 (time 1777576851.25 total 511.67 last 7.66) - open/close 8520 (time 1777576861.74 total 522.16 last 14.87) - open/close 8648 (time 1777576871.76 total 532.18 last 12.77) - open/close 8847 (time 1777576883.00 total 543.41 last 17.71) - open/close 9102 (time 1777576893.01 total 553.42 last 25.48) - open/close 9319 (time 1777576903.28 total 563.69 last 21.13) - open/close 9546 (time 1777576914.57 total 574.98 last 20.10) - open/close 9715 (time 1777576925.44 total 585.86 last 15.54) - open/close 9936 (time 1777576936.00 total 596.41 last 20.94) - open/close 10000 (time 1777576941.44 total 601.86 last 11.76) - open/close 10198 (time 1777576952.32 total 612.73 last 18.20) - open/close 10341 (time 1777576963.30 total 623.71 last 13.03) - open/close 10498 (time 1777576973.82 total 634.23 last 14.92) - open/close 10620 (time 1777576983.83 total 644.24 last 12.19) - open/close 10813 (time 1777576993.84 total 654.25 last 19.28) - open/close 11008 (time 1777577003.87 total 664.28 last 19.45) - open/close 11278 (time 1777577013.88 total 674.30 last 26.95) - open/close 11427 (time 1777577026.23 total 686.65 last 12.07) - open/close 11634 (time 1777577036.74 total 697.15 last 19.70) - open/close 11863 (time 1777577047.08 total 707.49 last 22.15) - open/close 12107 (time 1777577057.92 total 718.33 last 22.51) - open/close 12360 (time 1777577070.67 total 731.08 last 19.85) - open/close 12630 (time 1777577080.67 total 741.08 last 27.00) - open/close 12849 (time 1777577090.68 total 751.09 last 21.87) - open/close 13108 (time 1777577101.62 total 762.03 last 23.67) - open/close 13267 (time 1777577111.63 total 772.04 last 15.89) - open/close 13398 (time 1777577122.29 total 782.70 last 12.29) - open/close 13562 (time 1777577133.23 total 793.64 last 14.99) - open/close 13744 (time 1777577143.29 total 803.70 last 18.09) - open/close 13944 (time 1777577154.30 total 814.71 last 18.16) - open/close 14132 (time 1777577164.30 total 824.72 last 18.80) - open/close 14262 (time 1777577174.31 total 834.72 last 13.00) - open/close 14385 (time 1777577184.33 total 844.74 last 12.28) - open/close 14538 (time 1777577195.19 total 855.61 last 14.08) - open/close 14689 (time 1777577205.20 total 865.61 last 15.09) - open/close 14872 (time 1777577216.11 total 876.53 last 16.77) - open/close 15104 (time 1777577226.42 total 886.83 last 22.52) - open/close 15392 (time 1777577236.43 total 896.84 last 28.77) - open/close 15603 (time 1777577247.16 total 907.58 last 19.65) - open/close 15840 (time 1777577258.83 total 919.24 last 20.31) - open/close 16040 (time 1777577268.86 total 929.28 last 19.93) - open/close 16242 (time 1777577279.10 total 939.52 last 19.72) - open/close 16419 (time 1777577289.12 total 949.54 last 17.67) - open/close 16656 (time 1777577299.13 total 959.54 last 23.70) - open/close 16919 (time 1777577309.13 total 969.54 last 26.28) - open/close 17122 (time 1777577319.63 total 980.04 last 19.34) - open/close 17378 (time 1777577330.49 total 990.91 last 23.56) - open/close 17787 (time 1777577340.50 total 1000.91 last 40.88) - open/close 18438 (time 1777577350.50 total 1010.92 last 65.06) - open/close 19119 (time 1777577360.51 total 1020.92 last 68.09) - open/close 19811 (time 1777577370.51 total 1030.92 last 69.19) - open/close 20000 (time 1777577372.61 total 1033.03 last 89.65) - open/close 20643 (time 1777577382.62 total 1043.03 last 64.28) - open/close 21339 (time 1777577392.63 total 1053.04 last 69.54) - open/close 22042 (time 1777577402.64 total 1063.05 last 70.23) - open/close 22729 (time 1777577412.64 total 1073.05 last 68.70) - open/close 23350 (time 1777577422.66 total 1083.07 last 61.94) - open/close 23993 (time 1777577432.68 total 1093.09 last 64.18) - open/close 24583 (time 1777577442.68 total 1103.09 last 58.99) - open/close 25221 (time 1777577452.69 total 1113.10 last 63.76) - open/close 25829 (time 1777577462.69 total 1123.11 last 60.77) - open/close 26451 (time 1777577472.70 total 1133.11 last 62.17) - open/close 27103 (time 1777577482.70 total 1143.11 last 65.19) - open/close 27686 (time 1777577492.71 total 1153.12 last 58.26) - open/close 28300 (time 1777577502.71 total 1163.12 last 61.36) - open/close 28942 (time 1777577512.72 total 1173.13 last 64.14) - open/close 29588 (time 1777577522.72 total 1183.14 last 64.59) - open/close 30000 (time 1777577529.82 total 1190.23 last 58.07) - open/close 30650 (time 1777577539.83 total 1200.24 last 64.94) - open/close 31292 (time 1777577549.83 total 1210.25 last 64.17) - open/close 31998 (time 1777577559.84 total 1220.25 last 70.58) - open/close 32744 (time 1777577569.84 total 1230.25 last 74.60) - open/close 33441 (time 1777577579.84 total 1240.25 last 69.68) - open/close 34104 (time 1777577589.84 total 1250.26 last 66.27) - open/close 34836 (time 1777577599.86 total 1260.27 last 73.11) - open/close 35542 (time 1777577609.86 total 1270.27 last 70.55) - open/close 36112 (time 1777577619.88 total 1280.29 last 56.91) - open/close 36611 (time 1777577629.90 total 1290.31 last 49.80) - open/close 37180 (time 1777577639.90 total 1300.32 last 56.87) - open/close 37876 (time 1777577649.91 total 1310.32 last 69.58) - open/close 38563 (time 1777577659.91 total 1320.33 last 68.64) - open/close 39251 (time 1777577669.93 total 1330.34 last 68.69) - open/close 39888 (time 1777577679.94 total 1340.35 last 63.63) - open/close 40595 (time 1777577689.95 total 1350.36 last 70.67) - open/close 41264 (time 1777577699.95 total 1360.36 last 66.88) - open/close 41958 (time 1777577709.95 total 1370.36 last 69.38) - open/close 42673 (time 1777577719.96 total 1380.37 last 71.43) - open/close 43387 (time 1777577729.96 total 1390.38 last 71.38) - open/close 44120 (time 1777577739.97 total 1400.38 last 73.27) - open/close 44848 (time 1777577749.97 total 1410.38 last 72.77) - open/close 45574 (time 1777577759.98 total 1420.39 last 72.55) - open/close 46285 (time 1777577769.99 total 1430.40 last 71.04) - open/close 47026 (time 1777577780.00 total 1440.42 last 73.99) - open/close 47738 (time 1777577790.01 total 1450.42 last 71.17) - open/close 48449 (time 1777577800.01 total 1460.43 last 71.05) - open/close 49160 (time 1777577810.03 total 1470.44 last 71.00) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1479.34 seconds: 33.55 ops/second - unlinked 0 (time 1777577825 ; total 0 ; last 0) - unlinked 10000 (time 1777577980 ; total 155 ; last 155) - unlinked 20000 (time 1777578151 ; total 326 ; last 171) - unlinked 30000 (time 1777578307 ; total 482 ; last 156) - unlinked 40000 (time 1777578474 ; total 649 ; last 167) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 804 seconds: 61.731342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2