== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:57:43 (1787849863) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1787849991.86 total 10.94 last 13.44) - open/close 338 (time 1787850002.33 total 21.41 last 18.24) - open/close 515 (time 1787850012.86 total 31.94 last 16.81) - open/close 705 (time 1787850023.63 total 42.71 last 17.64) - open/close 886 (time 1787850033.64 total 52.72 last 18.08) - open/close 1129 (time 1787850043.69 total 62.76 last 24.19) - open/close 1253 (time 1787850054.35 total 73.43 last 11.63) - open/close 1348 (time 1787850064.73 total 83.80 last 9.16) - open/close 1528 (time 1787850075.73 total 94.80 last 16.36) - open/close 1752 (time 1787850085.73 total 104.80 last 22.40) - open/close 1882 (time 1787850095.74 total 114.81 last 12.99) - open/close 1978 (time 1787850105.92 total 124.99 last 9.43) - open/close 2101 (time 1787850115.92 total 135.00 last 12.30) - open/close 2315 (time 1787850126.41 total 145.49 last 20.40) - open/close 2394 (time 1787850136.58 total 155.65 last 7.77) - open/close 2497 (time 1787850147.36 total 166.43 last 9.56) - open/close 2600 (time 1787850160.15 total 179.22 last 8.05) - open/close 2666 (time 1787850171.20 total 190.28 last 5.97) - open/close 2933 (time 1787850181.52 total 200.60 last 25.88) - open/close 3086 (time 1787850192.40 total 211.48 last 14.06) - open/close 3231 (time 1787850202.42 total 221.49 last 14.47) - open/close 3443 (time 1787850212.76 total 231.83 last 20.51) - open/close 3571 (time 1787850224.64 total 243.71 last 10.78) - open/close 3783 (time 1787850235.73 total 254.81 last 19.10) - open/close 4019 (time 1787850246.75 total 265.83 last 21.42) - open/close 4242 (time 1787850259.20 total 278.27 last 17.91) - open/close 4398 (time 1787850270.17 total 289.24 last 14.23) - open/close 4508 (time 1787850280.18 total 299.25 last 10.99) - open/close 4672 (time 1787850290.68 total 309.75 last 15.61) - open/close 4813 (time 1787850300.69 total 319.76 last 14.09) - open/close 4936 (time 1787850310.70 total 329.77 last 12.29) - open/close 5151 (time 1787850321.49 total 340.56 last 19.91) - open/close 5366 (time 1787850332.44 total 351.51 last 19.64) - open/close 5523 (time 1787850343.44 total 362.51 last 14.27) - open/close 5694 (time 1787850354.51 total 373.58 last 15.45) - open/close 5787 (time 1787850364.81 total 383.88 last 9.03) - open/close 5935 (time 1787850377.70 total 396.77 last 11.48) - open/close 6170 (time 1787850387.72 total 406.80 last 23.44) - open/close 6270 (time 1787850400.47 total 419.54 last 7.85) - open/close 6397 (time 1787850412.12 total 431.19 last 10.90) - open/close 6579 (time 1787850422.17 total 441.24 last 18.11) - open/close 6781 (time 1787850434.95 total 454.03 last 15.80) - open/close 6822 (time 1787850445.52 total 464.59 last 3.88) - open/close 6904 (time 1787850455.84 total 474.92 last 7.94) - open/close 7128 (time 1787850466.85 total 485.92 last 20.36) - open/close 7254 (time 1787850476.87 total 495.95 last 12.57) - open/close 7357 (time 1787850486.90 total 505.98 last 10.27) - open/close 7522 (time 1787850497.74 total 516.81 last 15.23) - open/close 7668 (time 1787850507.76 total 526.83 last 14.57) - open/close 7848 (time 1787850517.81 total 536.88 last 17.91) - open/close 8034 (time 1787850528.13 total 547.21 last 18.01) - open/close 8265 (time 1787850538.16 total 557.24 last 23.03) - open/close 8460 (time 1787850548.18 total 567.25 last 19.47) - open/close 8614 (time 1787850558.22 total 577.29 last 15.34) - open/close 8782 (time 1787850568.63 total 587.71 last 16.13) - open/close 8841 (time 1787850578.66 total 597.73 last 5.89) - open/close 9000 (time 1787850588.69 total 607.76 last 15.85) - open/close 9146 (time 1787850598.70 total 617.77 last 14.59) - open/close 9215 (time 1787850608.70 total 627.77 last 6.90) - open/close 9394 (time 1787850619.41 total 638.49 last 16.71) - open/close 9497 (time 1787850629.97 total 649.04 last 9.76) - open/close 9673 (time 1787850641.75 total 660.82 last 14.94) - open/close 9759 (time 1787850653.98 total 673.05 last 7.03) - open/close 10000 (time 1787850663.93 total 683.00 last 24.23) - open/close 10129 (time 1787850673.94 total 693.01 last 12.88) - open/close 10350 (time 1787850684.77 total 703.84 last 20.40) - open/close 10497 (time 1787850695.37 total 714.44 last 13.87) - open/close 10660 (time 1787850705.39 total 724.46 last 16.27) - open/close 10850 (time 1787850715.80 total 734.88 last 18.24) - open/close 11012 (time 1787850725.81 total 744.88 last 16.19) - open/close 11129 (time 1787850736.46 total 755.53 last 10.99) - open/close 11247 (time 1787850746.46 total 765.53 last 11.80) - open/close 11440 (time 1787850756.89 total 775.96 last 18.51) - open/close 11584 (time 1787850767.89 total 786.96 last 13.09) - open/close 11803 (time 1787850778.12 total 797.19 last 21.40) - open/close 11879 (time 1787850788.30 total 807.37 last 7.47) - open/close 12063 (time 1787850798.80 total 817.87 last 17.52) - open/close 12232 (time 1787850808.97 total 828.04 last 16.62) - open/close 12406 (time 1787850820.03 total 839.10 last 15.74) - open/close 12542 (time 1787850830.22 total 849.29 last 13.34) - open/close 12696 (time 1787850841.00 total 860.08 last 14.28) - open/close 12754 (time 1787850851.03 total 870.10 last 5.79) - open/close 12872 (time 1787850861.40 total 880.47 last 11.38) - open/close 12961 (time 1787850872.80 total 891.88 last 7.80) - open/close 13049 (time 1787850883.15 total 902.23 last 8.50) - open/close 13251 (time 1787850893.83 total 912.91 last 18.92) - open/close 13427 (time 1787850903.85 total 922.93 last 17.57) - open/close 13621 (time 1787850914.07 total 933.14 last 18.99) - open/close 13689 (time 1787850924.07 total 943.14 last 6.80) - open/close 13824 (time 1787850934.29 total 953.36 last 13.22) - open/close 13936 (time 1787850944.85 total 963.92 last 10.60) - open/close 14063 (time 1787850955.71 total 974.78 last 11.70) - open/close 14193 (time 1787850966.33 total 985.40 last 12.24) - open/close 14335 (time 1787850976.66 total 995.73 last 13.75) - open/close 14464 (time 1787850987.35 total 1006.42 last 12.07) - open/close 14593 (time 1787850997.37 total 1016.44 last 12.87) - open/close 14807 (time 1787851008.14 total 1027.21 last 19.87) - open/close 15001 (time 1787851018.90 total 1037.97 last 18.03) - open/close 15219 (time 1787851028.91 total 1047.99 last 21.76) - open/close 15489 (time 1787851038.92 total 1058.00 last 26.97) - open/close 15701 (time 1787851048.93 total 1068.01 last 21.18) - open/close 15967 (time 1787851061.52 total 1080.59 last 21.14) - open/close 16164 (time 1787851071.52 total 1090.60 last 19.69) - open/close 16288 (time 1787851081.53 total 1100.60 last 12.39) - open/close 16494 (time 1787851091.54 total 1110.62 last 20.57) - open/close 17193 (time 1787851101.57 total 1120.65 last 69.68) - open/close 17859 (time 1787851111.59 total 1130.66 last 66.51) - open/close 18752 (time 1787851121.60 total 1140.67 last 89.23) - open/close 19531 (time 1787851131.60 total 1150.67 last 77.90) - open/close 20000 (time 1787851137.36 total 1156.43 last 81.40) - open/close 20947 (time 1787851147.36 total 1166.43 last 94.69) - open/close 21714 (time 1787851157.36 total 1176.44 last 76.65) - open/close 22527 (time 1787851167.39 total 1186.46 last 81.11) - open/close 23306 (time 1787851177.39 total 1196.46 last 77.90) - open/close 24110 (time 1787851187.39 total 1206.46 last 80.40) - open/close 24802 (time 1787851197.42 total 1216.49 last 69.00) - open/close 25565 (time 1787851207.42 total 1226.50 last 76.27) - open/close 26359 (time 1787851217.42 total 1236.50 last 79.38) - open/close 26977 (time 1787851227.44 total 1246.51 last 61.70) - open/close 27887 (time 1787851237.45 total 1256.52 last 90.96) - open/close 28695 (time 1787851247.45 total 1266.53 last 80.73) - open/close 29536 (time 1787851257.46 total 1276.54 last 84.02) - open/close 30000 (time 1787851263.38 total 1282.45 last 78.44) - open/close 30849 (time 1787851273.38 total 1292.46 last 84.86) - open/close 31563 (time 1787851283.39 total 1302.46 last 71.37) - open/close 32431 (time 1787851293.39 total 1312.46 last 86.80) - open/close 33210 (time 1787851303.39 total 1322.47 last 77.87) - open/close 33970 (time 1787851313.39 total 1332.47 last 75.98) - open/close 34576 (time 1787851323.44 total 1342.51 last 60.33) - open/close 35242 (time 1787851333.44 total 1352.52 last 66.56) - open/close 36134 (time 1787851343.45 total 1362.52 last 89.18) - open/close 37119 (time 1787851353.46 total 1372.53 last 98.36) - open/close 37999 (time 1787851363.47 total 1382.54 last 87.93) - open/close 38695 (time 1787851373.47 total 1392.55 last 69.56) - open/close 39329 (time 1787851383.51 total 1402.58 last 63.19) - open/close 39916 (time 1787851393.51 total 1412.59 last 58.66) - open/close 40449 (time 1787851403.52 total 1422.59 last 53.29) - open/close 41060 (time 1787851413.52 total 1432.60 last 61.04) - open/close 41702 (time 1787851423.53 total 1442.60 last 64.16) - open/close 42365 (time 1787851433.54 total 1452.61 last 66.23) - open/close 43080 (time 1787851443.54 total 1462.61 last 71.50) - open/close 43819 (time 1787851453.55 total 1472.62 last 73.85) - open/close 44599 (time 1787851463.55 total 1482.62 last 77.98) - open/close 45412 (time 1787851473.55 total 1492.62 last 81.30) - open/close 46298 (time 1787851483.56 total 1502.63 last 88.56) - open/close 47020 (time 1787851493.56 total 1512.63 last 72.16) - open/close 47578 (time 1787851503.57 total 1522.64 last 55.76) - open/close 48224 (time 1787851513.57 total 1532.64 last 64.59) - open/close 48953 (time 1787851523.57 total 1542.65 last 72.88) - open/close 49550 (time 1787851533.58 total 1552.65 last 59.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1554.02 seconds: 31.94 ops/second - unlinked 0 (time 1787851540 ; total 0 ; last 0) - unlinked 10000 (time 1787851688 ; total 148 ; last 148) - unlinked 20000 (time 1787851828 ; total 288 ; last 140) - unlinked 30000 (time 1787851980 ; total 440 ; last 152) - unlinked 40000 (time 1787852113 ; total 573 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 709 seconds: 70.001411 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2