== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:21:29 (1780330889) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 138 (time 1780331009.72 total 11.06 last 12.48) - open/close 310 (time 1780331020.55 total 21.89 last 15.88) - open/close 476 (time 1780331031.16 total 32.49 last 15.65) - open/close 618 (time 1780331041.48 total 42.81 last 13.76) - open/close 704 (time 1780331051.90 total 53.23 last 8.25) - open/close 824 (time 1780331062.47 total 63.80 last 11.35) - open/close 908 (time 1780331072.66 total 73.99 last 8.25) - open/close 997 (time 1780331083.14 total 84.47 last 8.49) - open/close 1146 (time 1780331093.18 total 94.52 last 14.84) - open/close 1331 (time 1780331105.15 total 106.48 last 15.46) - open/close 1469 (time 1780331115.99 total 117.32 last 12.73) - open/close 1584 (time 1780331126.35 total 127.68 last 11.10) - open/close 1721 (time 1780331136.73 total 138.06 last 13.20) - open/close 1923 (time 1780331146.74 total 148.07 last 20.19) - open/close 2008 (time 1780331156.74 total 158.08 last 8.49) - open/close 2227 (time 1780331166.76 total 168.09 last 21.86) - open/close 2472 (time 1780331177.61 total 178.94 last 22.58) - open/close 2616 (time 1780331187.61 total 188.95 last 14.39) - open/close 2897 (time 1780331197.64 total 198.98 last 28.02) - open/close 3030 (time 1780331208.61 total 209.94 last 12.13) - open/close 3218 (time 1780331218.94 total 220.27 last 18.20) - open/close 3400 (time 1780331229.90 total 231.24 last 16.60) - open/close 3502 (time 1780331240.53 total 241.86 last 9.60) - open/close 3639 (time 1780331250.56 total 251.89 last 13.65) - open/close 3766 (time 1780331261.12 total 262.45 last 12.03) - open/close 4019 (time 1780331271.13 total 272.46 last 25.27) - open/close 4251 (time 1780331281.48 total 282.81 last 22.41) - open/close 4529 (time 1780331291.48 total 292.82 last 27.79) - open/close 4791 (time 1780331301.48 total 302.82 last 26.20) - open/close 5057 (time 1780331311.55 total 312.89 last 26.41) - open/close 5300 (time 1780331321.56 total 322.89 last 24.29) - open/close 5497 (time 1780331332.61 total 333.94 last 17.83) - open/close 5590 (time 1780331342.66 total 344.00 last 9.25) - open/close 5837 (time 1780331352.68 total 354.01 last 24.66) - open/close 6011 (time 1780331363.12 total 364.46 last 16.66) - open/close 6268 (time 1780331373.13 total 374.46 last 25.69) - open/close 6388 (time 1780331383.36 total 384.70 last 11.72) - open/close 6669 (time 1780331393.36 total 394.70 last 28.09) - open/close 6790 (time 1780331403.46 total 404.80 last 11.98) - open/close 7014 (time 1780331413.47 total 414.80 last 22.39) - open/close 7275 (time 1780331423.49 total 424.83 last 26.03) - open/close 7545 (time 1780331433.50 total 434.83 last 26.99) - open/close 7694 (time 1780331443.99 total 445.32 last 14.20) - open/close 7956 (time 1780331454.93 total 456.26 last 23.95) - open/close 8222 (time 1780331465.60 total 466.93 last 24.93) - open/close 8485 (time 1780331475.60 total 476.94 last 26.29) - open/close 8579 (time 1780331485.90 total 487.23 last 9.13) - open/close 8784 (time 1780331495.90 total 497.23 last 20.50) - open/close 8963 (time 1780331505.92 total 507.26 last 17.86) - open/close 9163 (time 1780331516.56 total 517.89 last 18.81) - open/close 9308 (time 1780331527.28 total 528.62 last 13.52) - open/close 9416 (time 1780331537.31 total 538.64 last 10.78) - open/close 9562 (time 1780331550.09 total 551.42 last 11.42) - open/close 9868 (time 1780331560.13 total 561.46 last 30.47) - open/close 9948 (time 1780331570.18 total 571.52 last 7.96) - open/close 10000 (time 1780331573.26 total 574.60 last 16.89) - open/close 10158 (time 1780331583.29 total 584.62 last 15.77) - open/close 10301 (time 1780331593.31 total 594.64 last 14.27) - open/close 10404 (time 1780331603.78 total 605.12 last 9.83) - open/close 10641 (time 1780331614.28 total 615.61 last 22.59) - open/close 10708 (time 1780331625.15 total 626.49 last 6.16) - open/close 10803 (time 1780331635.78 total 637.11 last 8.94) - open/close 10928 (time 1780331645.81 total 647.14 last 12.47) - open/close 10998 (time 1780331655.81 total 657.14 last 7.00) - open/close 11038 (time 1780331666.21 total 667.54 last 3.85) - open/close 11134 (time 1780331676.69 total 678.03 last 9.15) - open/close 11268 (time 1780331686.87 total 688.20 last 13.17) - open/close 11473 (time 1780331697.11 total 698.44 last 20.03) - open/close 11630 (time 1780331707.13 total 708.46 last 15.67) - open/close 11800 (time 1780331717.14 total 718.48 last 16.97) - open/close 12035 (time 1780331727.15 total 728.49 last 23.48) - open/close 12180 (time 1780331737.16 total 738.49 last 14.50) - open/close 12367 (time 1780331748.03 total 749.36 last 17.21) - open/close 12584 (time 1780331758.98 total 760.32 last 19.80) - open/close 12642 (time 1780331771.84 total 773.18 last 4.51) - open/close 12819 (time 1780331782.42 total 783.76 last 16.73) - open/close 13017 (time 1780331793.28 total 794.62 last 18.23) - open/close 13199 (time 1780331803.30 total 804.63 last 18.18) - open/close 13418 (time 1780331813.89 total 815.22 last 20.67) - open/close 13629 (time 1780331823.96 total 825.30 last 20.94) - open/close 13717 (time 1780331833.96 total 835.30 last 8.80) - open/close 13785 (time 1780331844.80 total 846.13 last 6.28) - open/close 13974 (time 1780331854.82 total 856.15 last 18.86) - open/close 14108 (time 1780331864.82 total 866.16 last 13.40) - open/close 14344 (time 1780331875.14 total 876.47 last 22.87) - open/close 14464 (time 1780331885.15 total 886.48 last 11.99) - open/close 14638 (time 1780331896.19 total 897.53 last 15.76) - open/close 14761 (time 1780331907.22 total 908.55 last 11.16) - open/close 14989 (time 1780331917.71 total 919.04 last 21.74) - open/close 15218 (time 1780331928.05 total 929.39 last 22.13) - open/close 15478 (time 1780331941.19 total 942.52 last 19.80) - open/close 15679 (time 1780331951.44 total 952.78 last 19.60) - open/close 15844 (time 1780331961.87 total 963.21 last 15.82) - open/close 15998 (time 1780331971.96 total 973.30 last 15.26) - open/close 16147 (time 1780331981.97 total 983.30 last 14.89) - open/close 16361 (time 1780331992.09 total 993.42 last 21.15) - open/close 16547 (time 1780332002.10 total 1003.43 last 18.59) - open/close 16629 (time 1780332012.24 total 1013.58 last 8.08) - open/close 16832 (time 1780332023.24 total 1024.57 last 18.46) - open/close 16914 (time 1780332033.25 total 1034.58 last 8.19) - open/close 17082 (time 1780332043.47 total 1044.81 last 16.43) - open/close 17346 (time 1780332053.79 total 1055.13 last 25.58) - open/close 17486 (time 1780332066.00 total 1067.33 last 11.47) - open/close 17655 (time 1780332077.25 total 1078.59 last 15.02) - open/close 18271 (time 1780332087.27 total 1088.60 last 61.51) - open/close 18948 (time 1780332097.27 total 1098.60 last 67.69) - open/close 19819 (time 1780332107.29 total 1108.62 last 86.95) - open/close 20000 (time 1780332109.62 total 1110.95 last 77.58) - open/close 20720 (time 1780332119.62 total 1120.96 last 71.98) - open/close 21610 (time 1780332129.63 total 1130.96 last 88.94) - open/close 22391 (time 1780332139.64 total 1140.97 last 78.04) - open/close 23238 (time 1780332149.64 total 1150.98 last 84.62) - open/close 24136 (time 1780332159.65 total 1160.99 last 89.74) - open/close 24984 (time 1780332169.67 total 1171.00 last 84.65) - open/close 25896 (time 1780332179.68 total 1181.01 last 91.11) - open/close 26921 (time 1780332189.68 total 1191.01 last 102.49) - open/close 27699 (time 1780332199.70 total 1201.03 last 77.68) - open/close 28438 (time 1780332209.70 total 1211.03 last 73.87) - open/close 29319 (time 1780332219.70 total 1221.04 last 88.08) - open/close 30000 (time 1780332228.12 total 1229.45 last 80.92) - open/close 30766 (time 1780332238.12 total 1239.46 last 76.57) - open/close 31543 (time 1780332248.14 total 1249.47 last 77.55) - open/close 32449 (time 1780332258.14 total 1259.47 last 90.60) - open/close 33242 (time 1780332268.14 total 1269.47 last 79.30) - open/close 33999 (time 1780332278.15 total 1279.49 last 75.61) - open/close 34785 (time 1780332288.16 total 1289.50 last 78.53) - open/close 35643 (time 1780332298.18 total 1299.52 last 85.61) - open/close 36484 (time 1780332308.19 total 1309.52 last 84.06) - open/close 37217 (time 1780332318.22 total 1319.55 last 73.07) - open/close 38121 (time 1780332328.23 total 1329.56 last 90.35) - open/close 38931 (time 1780332338.23 total 1339.56 last 81.00) - open/close 39796 (time 1780332348.23 total 1349.57 last 86.44) - open/close 40000 (time 1780332350.53 total 1351.87 last 88.61) - open/close 40831 (time 1780332360.54 total 1361.88 last 83.04) - open/close 41711 (time 1780332370.54 total 1371.88 last 87.98) - open/close 42627 (time 1780332380.56 total 1381.89 last 91.50) - open/close 43577 (time 1780332390.56 total 1391.89 last 94.95) - open/close 44590 (time 1780332400.56 total 1401.89 last 101.30) - open/close 45345 (time 1780332410.57 total 1411.91 last 75.42) - open/close 46306 (time 1780332420.57 total 1421.91 last 96.07) - open/close 47157 (time 1780332430.58 total 1431.92 last 85.04) - open/close 47970 (time 1780332440.59 total 1441.92 last 81.26) - open/close 48822 (time 1780332450.59 total 1451.93 last 85.16) - open/close 49523 (time 1780332460.60 total 1461.94 last 70.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1463.50 seconds: 33.91 ops/second - unlinked 0 (time 1780332467 ; total 0 ; last 0) - unlinked 10000 (time 1780332590 ; total 123 ; last 123) - unlinked 20000 (time 1780332716 ; total 249 ; last 126) - unlinked 30000 (time 1780332848 ; total 381 ; last 132) - unlinked 40000 (time 1780332967 ; total 500 ; last 119) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 628 seconds: 79.030258 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2