== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:18:42 (1779131922) start mds service on oleg141-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg141-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1779132041.56 total 10.62 last 8.95) - open/close 172 (time 1779132051.56 total 20.62 last 7.70) - open/close 307 (time 1779132061.59 total 30.65 last 13.46) - open/close 574 (time 1779132072.41 total 41.47 last 24.68) - open/close 817 (time 1779132083.07 total 52.14 last 22.78) - open/close 998 (time 1779132093.83 total 62.90 last 16.82) - open/close 1114 (time 1779132106.84 total 75.91 last 8.92) - open/close 1200 (time 1779132116.89 total 85.95 last 8.56) - open/close 1382 (time 1779132127.83 total 96.90 last 16.63) - open/close 1633 (time 1779132138.19 total 107.25 last 24.24) - open/close 1871 (time 1779132149.01 total 118.07 last 22.00) - open/close 2099 (time 1779132160.08 total 129.14 last 20.59) - open/close 2325 (time 1779132170.77 total 139.83 last 21.14) - open/close 2650 (time 1779132181.71 total 150.78 last 29.70) - open/close 2880 (time 1779132191.84 total 160.90 last 22.72) - open/close 3017 (time 1779132201.85 total 170.91 last 13.68) - open/close 3277 (time 1779132212.42 total 181.48 last 24.60) - open/close 3482 (time 1779132223.49 total 192.56 last 18.51) - open/close 3741 (time 1779132234.26 total 203.32 last 24.07) - open/close 3930 (time 1779132244.55 total 213.61 last 18.36) - open/close 4078 (time 1779132254.75 total 223.81 last 14.51) - open/close 4249 (time 1779132264.90 total 233.97 last 16.83) - open/close 4497 (time 1779132276.04 total 245.11 last 22.26) - open/close 4691 (time 1779132286.29 total 255.35 last 18.94) - open/close 4814 (time 1779132297.28 total 266.34 last 11.19) - open/close 5010 (time 1779132307.34 total 276.40 last 19.48) - open/close 5100 (time 1779132317.42 total 286.48 last 8.93) - open/close 5258 (time 1779132328.47 total 297.53 last 14.30) - open/close 5397 (time 1779132339.79 total 308.85 last 12.28) - open/close 5567 (time 1779132350.11 total 319.17 last 16.47) - open/close 5718 (time 1779132360.61 total 329.67 last 14.38) - open/close 5948 (time 1779132371.53 total 340.59 last 21.05) - open/close 6145 (time 1779132383.96 total 353.03 last 15.85) - open/close 6391 (time 1779132394.44 total 363.50 last 23.48) - open/close 6499 (time 1779132406.12 total 375.19 last 9.24) - open/close 6606 (time 1779132416.14 total 385.20 last 10.69) - open/close 6720 (time 1779132426.14 total 395.20 last 11.40) - open/close 6781 (time 1779132437.02 total 406.08 last 5.61) - open/close 6997 (time 1779132447.04 total 416.10 last 21.56) - open/close 7099 (time 1779132457.49 total 426.55 last 9.76) - open/close 7245 (time 1779132469.55 total 438.62 last 12.10) - open/close 7384 (time 1779132479.92 total 448.98 last 13.41) - open/close 7460 (time 1779132490.13 total 459.19 last 7.45) - open/close 7639 (time 1779132500.13 total 469.19 last 17.90) - open/close 7760 (time 1779132511.20 total 480.27 last 10.93) - open/close 7966 (time 1779132521.93 total 491.00 last 19.20) - open/close 8138 (time 1779132531.96 total 501.02 last 17.16) - open/close 8394 (time 1779132541.96 total 511.02 last 25.59) - open/close 8575 (time 1779132552.53 total 521.59 last 17.13) - open/close 8711 (time 1779132562.53 total 531.59 last 13.60) - open/close 8782 (time 1779132572.55 total 541.61 last 7.09) - open/close 8925 (time 1779132583.50 total 552.56 last 13.06) - open/close 9026 (time 1779132594.13 total 563.19 last 9.50) - open/close 9099 (time 1779132604.14 total 573.20 last 7.29) - open/close 9193 (time 1779132614.61 total 583.67 last 8.98) - open/close 9303 (time 1779132627.46 total 596.52 last 8.56) - open/close 9383 (time 1779132637.79 total 606.85 last 7.75) - open/close 9506 (time 1779132647.80 total 616.86 last 12.29) - open/close 9572 (time 1779132657.80 total 626.87 last 6.60) - open/close 9654 (time 1779132667.84 total 636.91 last 8.17) - open/close 9822 (time 1779132677.86 total 646.92 last 16.77) - open/close 9926 (time 1779132688.65 total 657.71 last 9.64) - open/close 10000 (time 1779132691.32 total 660.38 last 27.70) - open/close 10158 (time 1779132701.32 total 670.39 last 15.79) - open/close 10304 (time 1779132711.77 total 680.83 last 13.97) - open/close 10470 (time 1779132721.79 total 690.86 last 16.56) - open/close 10594 (time 1779132731.81 total 700.87 last 12.38) - open/close 10687 (time 1779132742.69 total 711.75 last 8.55) - open/close 10819 (time 1779132753.18 total 722.24 last 12.58) - open/close 10938 (time 1779132763.30 total 732.36 last 11.76) - open/close 11114 (time 1779132773.54 total 742.61 last 17.18) - open/close 11357 (time 1779132783.76 total 752.83 last 23.77) - open/close 11477 (time 1779132793.80 total 762.87 last 11.95) - open/close 11546 (time 1779132804.63 total 773.69 last 6.37) - open/close 11698 (time 1779132814.65 total 783.71 last 15.18) - open/close 11835 (time 1779132825.48 total 794.54 last 12.65) - open/close 11904 (time 1779132835.73 total 804.79 last 6.73) - open/close 11963 (time 1779132845.75 total 814.81 last 5.89) - open/close 12126 (time 1779132855.78 total 824.84 last 16.24) - open/close 12228 (time 1779132866.52 total 835.58 last 9.50) - open/close 12335 (time 1779132877.20 total 846.27 last 10.01) - open/close 12482 (time 1779132887.21 total 856.27 last 14.69) - open/close 12578 (time 1779132898.00 total 867.07 last 8.89) - open/close 12647 (time 1779132908.40 total 877.46 last 6.64) - open/close 12826 (time 1779132918.92 total 887.99 last 17.01) - open/close 12989 (time 1779132928.94 total 898.00 last 16.28) - open/close 13121 (time 1779132939.81 total 908.87 last 12.14) - open/close 13250 (time 1779132951.08 total 920.14 last 11.45) - open/close 13345 (time 1779132961.74 total 930.81 last 8.91) - open/close 13481 (time 1779132971.99 total 941.05 last 13.27) - open/close 13605 (time 1779132982.00 total 951.07 last 12.38) - open/close 13760 (time 1779132992.03 total 961.09 last 15.46) - open/close 13908 (time 1779133005.03 total 974.09 last 11.38) - open/close 14077 (time 1779133015.04 total 984.10 last 16.89) - open/close 14167 (time 1779133025.63 total 994.69 last 8.50) - open/close 14398 (time 1779133035.98 total 1005.05 last 22.31) - open/close 14510 (time 1779133046.05 total 1015.11 last 11.13) - open/close 14588 (time 1779133056.40 total 1025.46 last 7.54) - open/close 14737 (time 1779133066.42 total 1035.48 last 14.88) - open/close 14895 (time 1779133079.33 total 1048.39 last 12.23) - open/close 15046 (time 1779133090.13 total 1059.19 last 13.99) - open/close 15219 (time 1779133100.13 total 1069.19 last 17.30) - open/close 15381 (time 1779133111.12 total 1080.19 last 14.73) - open/close 15516 (time 1779133121.17 total 1090.23 last 13.44) - open/close 15824 (time 1779133131.19 total 1100.25 last 30.74) - open/close 16466 (time 1779133141.20 total 1110.27 last 64.09) - open/close 17131 (time 1779133151.23 total 1120.29 last 66.36) - open/close 17615 (time 1779133161.23 total 1130.30 last 48.36) - open/close 18267 (time 1779133171.24 total 1140.30 last 65.19) - open/close 18855 (time 1779133181.25 total 1150.31 last 58.74) - open/close 19472 (time 1779133191.26 total 1160.32 last 61.63) - open/close 20000 (time 1779133199.17 total 1168.23 last 66.75) - open/close 20478 (time 1779133209.17 total 1178.23 last 47.79) - open/close 21055 (time 1779133219.18 total 1188.24 last 57.67) - open/close 21635 (time 1779133229.18 total 1198.25 last 57.97) - open/close 22280 (time 1779133239.19 total 1208.25 last 64.45) - open/close 22810 (time 1779133249.20 total 1218.26 last 52.96) - open/close 23479 (time 1779133259.21 total 1228.27 last 66.85) - open/close 24148 (time 1779133269.21 total 1238.27 last 66.88) - open/close 24805 (time 1779133279.22 total 1248.28 last 65.67) - open/close 25386 (time 1779133289.24 total 1258.30 last 57.98) - open/close 26046 (time 1779133299.24 total 1268.30 last 65.98) - open/close 26777 (time 1779133309.25 total 1278.31 last 73.05) - open/close 27453 (time 1779133319.25 total 1288.32 last 67.55) - open/close 28051 (time 1779133329.28 total 1298.34 last 59.67) - open/close 28698 (time 1779133339.28 total 1308.34 last 64.66) - open/close 29365 (time 1779133349.32 total 1318.38 last 66.47) - open/close 29796 (time 1779133359.32 total 1328.39 last 43.07) - open/close 30000 (time 1779133363.28 total 1332.34 last 51.61) - open/close 30636 (time 1779133373.28 total 1342.34 last 63.59) - open/close 31237 (time 1779133383.30 total 1352.36 last 59.96) - open/close 31710 (time 1779133393.31 total 1362.37 last 47.28) - open/close 32281 (time 1779133403.31 total 1372.37 last 57.07) - open/close 32738 (time 1779133413.31 total 1382.37 last 45.70) - open/close 33285 (time 1779133423.34 total 1392.40 last 54.54) - open/close 33706 (time 1779133433.35 total 1402.41 last 42.06) - open/close 34316 (time 1779133443.36 total 1412.42 last 60.95) - open/close 34855 (time 1779133453.37 total 1422.43 last 53.85) - open/close 35516 (time 1779133463.39 total 1432.46 last 65.93) - open/close 36215 (time 1779133473.39 total 1442.46 last 69.89) - open/close 36848 (time 1779133483.43 total 1452.49 last 63.10) - open/close 37531 (time 1779133493.43 total 1462.50 last 68.25) - open/close 38267 (time 1779133503.44 total 1472.50 last 73.54) - open/close 38916 (time 1779133513.46 total 1482.52 last 64.77) - open/close 39495 (time 1779133523.47 total 1492.53 last 57.86) - open/close 40000 (time 1779133531.21 total 1500.28 last 65.20) - open/close 40658 (time 1779133541.22 total 1510.28 last 65.76) - open/close 41318 (time 1779133551.23 total 1520.29 last 65.97) - open/close 41823 (time 1779133561.23 total 1530.29 last 50.47) - open/close 42402 (time 1779133571.25 total 1540.31 last 57.82) - open/close 42954 (time 1779133581.25 total 1550.32 last 55.15) - open/close 43638 (time 1779133591.25 total 1560.32 last 68.39) - open/close 44319 (time 1779133601.27 total 1570.33 last 68.02) - open/close 44904 (time 1779133611.28 total 1580.34 last 58.42) - open/close 45560 (time 1779133621.28 total 1590.34 last 65.59) - open/close 46262 (time 1779133631.28 total 1600.35 last 70.19) - open/close 46911 (time 1779133641.31 total 1610.37 last 64.75) - open/close 47662 (time 1779133651.31 total 1620.38 last 75.05) - open/close 48482 (time 1779133661.31 total 1630.38 last 81.99) - open/close 49064 (time 1779133671.32 total 1640.38 last 58.17) - open/close 49617 (time 1779133681.33 total 1650.39 last 55.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1650.54 seconds: 30.07 ops/second - unlinked 0 (time 1779133687 ; total 0 ; last 0) - unlinked 10000 (time 1779133847 ; total 160 ; last 160) - unlinked 20000 (time 1779134004 ; total 317 ; last 157) - unlinked 30000 (time 1779134154 ; total 467 ; last 150) - unlinked 40000 (time 1779134313 ; total 626 ; last 159) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 774 seconds: 64.122742 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2