== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:20:27 (1787829627) start mds service on oleg427-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1787829712.43 total 10.02 last 20.75) - open/close 371 (time 1787829722.88 total 20.48 last 15.59) - open/close 499 (time 1787829733.87 total 31.46 last 11.66) - open/close 560 (time 1787829744.47 total 42.06 last 5.75) - open/close 711 (time 1787829756.11 total 53.71 last 12.96) - open/close 930 (time 1787829766.14 total 63.74 last 21.84) - open/close 1007 (time 1787829776.14 total 73.74 last 7.70) - open/close 1132 (time 1787829786.95 total 84.54 last 11.57) - open/close 1242 (time 1787829796.97 total 94.57 last 10.97) - open/close 1326 (time 1787829809.15 total 106.74 last 6.90) - open/close 1600 (time 1787829820.10 total 117.70 last 25.01) - open/close 1813 (time 1787829831.11 total 128.70 last 19.36) - open/close 1943 (time 1787829842.07 total 139.66 last 11.86) - open/close 2205 (time 1787829852.59 total 150.19 last 24.89) - open/close 2464 (time 1787829862.84 total 160.43 last 25.29) - open/close 2737 (time 1787829872.95 total 170.55 last 26.98) - open/close 2955 (time 1787829883.38 total 180.97 last 20.91) - open/close 3237 (time 1787829893.40 total 191.00 last 28.14) - open/close 3412 (time 1787829907.01 total 204.60 last 12.86) - open/close 3655 (time 1787829917.30 total 214.90 last 23.61) - open/close 3884 (time 1787829927.30 total 224.90 last 22.89) - open/close 3949 (time 1787829937.91 total 235.50 last 6.13) - open/close 4280 (time 1787829948.47 total 246.07 last 31.33) - open/close 4370 (time 1787829958.48 total 256.08 last 8.99) - open/close 4473 (time 1787829969.12 total 266.71 last 9.69) - open/close 4712 (time 1787829979.13 total 276.72 last 23.88) - open/close 4866 (time 1787829989.30 total 286.90 last 15.13) - open/close 5232 (time 1787829999.31 total 296.91 last 36.58) - open/close 5372 (time 1787830009.60 total 307.20 last 13.61) - open/close 5480 (time 1787830019.62 total 317.22 last 10.78) - open/close 5644 (time 1787830030.46 total 328.06 last 15.13) - open/close 5778 (time 1787830040.66 total 338.25 last 13.15) - open/close 5939 (time 1787830051.26 total 348.86 last 15.18) - open/close 6016 (time 1787830062.08 total 359.67 last 7.12) - open/close 6246 (time 1787830073.08 total 370.68 last 20.90) - open/close 6366 (time 1787830084.09 total 381.68 last 10.90) - open/close 6536 (time 1787830094.21 total 391.81 last 16.79) - open/close 6707 (time 1787830104.21 total 401.81 last 17.09) - open/close 6912 (time 1787830114.75 total 412.34 last 19.47) - open/close 7070 (time 1787830125.63 total 423.22 last 14.52) - open/close 7236 (time 1787830135.63 total 433.23 last 16.59) - open/close 7435 (time 1787830145.78 total 443.37 last 19.62) - open/close 7596 (time 1787830156.71 total 454.31 last 14.72) - open/close 7811 (time 1787830166.73 total 464.32 last 21.48) - open/close 7929 (time 1787830178.41 total 476.01 last 10.10) - open/close 8084 (time 1787830188.99 total 486.58 last 14.66) - open/close 8378 (time 1787830199.81 total 497.41 last 27.16) - open/close 8643 (time 1787830209.83 total 507.43 last 26.45) - open/close 8867 (time 1787830220.41 total 518.00 last 21.18) - open/close 9108 (time 1787830230.41 total 528.01 last 24.09) - open/close 9316 (time 1787830241.76 total 539.35 last 18.33) - open/close 9584 (time 1787830254.20 total 551.79 last 21.54) - open/close 9792 (time 1787830264.44 total 562.03 last 20.31) - open/close 9957 (time 1787830274.93 total 572.53 last 15.72) - open/close 10000 (time 1787830278.07 total 575.67 last 13.70) - open/close 10112 (time 1787830288.95 total 586.54 last 10.30) - open/close 10332 (time 1787830298.96 total 596.55 last 21.98) - open/close 10529 (time 1787830309.08 total 606.67 last 19.47) - open/close 10644 (time 1787830319.62 total 617.22 last 10.91) - open/close 10704 (time 1787830331.32 total 628.92 last 5.13) - open/close 10824 (time 1787830341.36 total 638.96 last 11.95) - open/close 10910 (time 1787830351.73 total 649.33 last 8.29) - open/close 11071 (time 1787830361.98 total 659.58 last 15.71) - open/close 11294 (time 1787830372.88 total 670.47 last 20.47) - open/close 11473 (time 1787830382.88 total 680.47 last 17.90) - open/close 11634 (time 1787830393.27 total 690.86 last 15.50) - open/close 11715 (time 1787830403.38 total 700.97 last 8.01) - open/close 11847 (time 1787830414.54 total 712.14 last 11.83) - open/close 12010 (time 1787830424.55 total 722.15 last 16.28) - open/close 12136 (time 1787830434.88 total 732.48 last 12.20) - open/close 12210 (time 1787830445.07 total 742.67 last 7.26) - open/close 12370 (time 1787830455.74 total 753.33 last 15.00) - open/close 12551 (time 1787830466.17 total 763.77 last 17.34) - open/close 12688 (time 1787830477.13 total 774.73 last 12.50) - open/close 12867 (time 1787830487.15 total 784.75 last 17.86) - open/close 13077 (time 1787830497.16 total 794.75 last 20.99) - open/close 13192 (time 1787830507.18 total 804.77 last 11.48) - open/close 13295 (time 1787830518.19 total 815.79 last 9.35) - open/close 13407 (time 1787830528.21 total 825.80 last 11.19) - open/close 13531 (time 1787830538.23 total 835.83 last 12.37) - open/close 13613 (time 1787830548.72 total 846.32 last 7.82) - open/close 13732 (time 1787830558.73 total 856.32 last 11.90) - open/close 13877 (time 1787830568.74 total 866.33 last 14.48) - open/close 14069 (time 1787830578.74 total 876.33 last 19.20) - open/close 14179 (time 1787830588.75 total 886.34 last 10.99) - open/close 14290 (time 1787830598.77 total 896.37 last 11.07) - open/close 14413 (time 1787830609.80 total 907.40 last 11.15) - open/close 14478 (time 1787830620.08 total 917.68 last 6.32) - open/close 14603 (time 1787830630.48 total 928.08 last 12.02) - open/close 14715 (time 1787830641.09 total 938.69 last 10.55) - open/close 14843 (time 1787830651.44 total 949.04 last 12.37) - open/close 14968 (time 1787830661.63 total 959.23 last 12.27) - open/close 15193 (time 1787830671.66 total 969.25 last 22.44) - open/close 15377 (time 1787830682.62 total 980.21 last 16.79) - open/close 15626 (time 1787830692.62 total 990.22 last 24.89) - open/close 15820 (time 1787830702.65 total 1000.24 last 19.36) - open/close 15974 (time 1787830713.54 total 1011.14 last 14.13) - open/close 16085 (time 1787830724.59 total 1022.19 last 10.05) - open/close 16252 (time 1787830735.58 total 1033.17 last 15.20) - open/close 16310 (time 1787830745.60 total 1043.20 last 5.79) - open/close 16496 (time 1787830755.61 total 1053.21 last 18.58) - open/close 16684 (time 1787830765.61 total 1063.21 last 18.80) - open/close 16824 (time 1787830775.62 total 1073.22 last 13.99) - open/close 17291 (time 1787830785.63 total 1083.23 last 46.64) - open/close 17936 (time 1787830795.67 total 1093.26 last 64.29) - open/close 18603 (time 1787830805.67 total 1103.27 last 66.66) - open/close 19270 (time 1787830815.68 total 1113.28 last 66.65) - open/close 19966 (time 1787830825.68 total 1123.28 last 69.59) - open/close 20699 (time 1787830835.69 total 1133.28 last 73.27) - open/close 21395 (time 1787830845.70 total 1143.30 last 69.48) - open/close 22125 (time 1787830855.72 total 1153.31 last 72.90) - open/close 22770 (time 1787830865.73 total 1163.32 last 64.44) - open/close 23467 (time 1787830875.74 total 1173.33 last 69.63) - open/close 24157 (time 1787830885.75 total 1183.34 last 68.93) - open/close 24873 (time 1787830895.75 total 1193.35 last 71.58) - open/close 25529 (time 1787830905.76 total 1203.35 last 65.57) - open/close 26242 (time 1787830915.76 total 1213.35 last 71.30) - open/close 26861 (time 1787830925.77 total 1223.36 last 61.84) - open/close 27516 (time 1787830935.77 total 1233.37 last 65.46) - open/close 28103 (time 1787830945.79 total 1243.38 last 58.61) - open/close 28765 (time 1787830955.79 total 1253.38 last 66.20) - open/close 29478 (time 1787830965.79 total 1263.39 last 71.27) - open/close 30000 (time 1787830973.03 total 1270.63 last 72.09) - open/close 30727 (time 1787830983.04 total 1280.63 last 72.65) - open/close 31467 (time 1787830993.05 total 1290.64 last 73.95) - open/close 32193 (time 1787831003.06 total 1300.65 last 72.50) - open/close 32865 (time 1787831013.08 total 1310.68 last 67.04) - open/close 33601 (time 1787831023.09 total 1320.69 last 73.54) - open/close 34360 (time 1787831033.09 total 1330.69 last 75.89) - open/close 35088 (time 1787831043.10 total 1340.69 last 72.78) - open/close 35800 (time 1787831053.12 total 1350.71 last 71.03) - open/close 36503 (time 1787831063.12 total 1360.72 last 70.27) - open/close 37142 (time 1787831073.14 total 1370.73 last 63.82) - open/close 37683 (time 1787831083.15 total 1380.75 last 54.03) - open/close 38364 (time 1787831093.17 total 1390.76 last 67.98) - open/close 38977 (time 1787831103.18 total 1400.77 last 61.25) - open/close 39624 (time 1787831113.18 total 1410.78 last 64.69) - open/close 40000 (time 1787831119.08 total 1416.67 last 63.77) - open/close 40672 (time 1787831129.08 total 1426.68 last 67.16) - open/close 41357 (time 1787831139.08 total 1436.68 last 68.50) - open/close 42081 (time 1787831149.09 total 1446.69 last 72.34) - open/close 42802 (time 1787831159.09 total 1456.69 last 72.07) - open/close 43520 (time 1787831169.10 total 1466.70 last 71.75) - open/close 44212 (time 1787831179.11 total 1476.71 last 69.14) - open/close 44885 (time 1787831189.11 total 1486.71 last 67.30) - open/close 45621 (time 1787831199.12 total 1496.72 last 73.53) - open/close 46308 (time 1787831209.13 total 1506.72 last 68.64) - open/close 46996 (time 1787831219.14 total 1516.73 last 68.75) - open/close 47649 (time 1787831229.14 total 1526.74 last 65.24) - open/close 48277 (time 1787831239.15 total 1536.75 last 62.75) - open/close 48957 (time 1787831249.16 total 1546.75 last 67.98) - open/close 49616 (time 1787831259.16 total 1556.76 last 65.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 1558.59 seconds: 31.91 ops/second - unlinked 0 (time 1787831265 ; total 0 ; last 0) - unlinked 10000 (time 1787831423 ; total 158 ; last 158) - unlinked 20000 (time 1787831575 ; total 310 ; last 152) - unlinked 30000 (time 1787831735 ; total 470 ; last 160) - unlinked 40000 (time 1787831890 ; total 625 ; last 155) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 773 seconds: 64.332474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2