== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:54:50 (1780336490) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1780336596.21 total 10.57 last 12.77) - open/close 354 (time 1780336607.03 total 21.39 last 20.25) - open/close 557 (time 1780336617.52 total 31.87 last 19.36) - open/close 746 (time 1780336627.72 total 42.08 last 18.53) - open/close 875 (time 1780336638.40 total 52.76 last 12.08) - open/close 1137 (time 1780336648.50 total 62.86 last 25.94) - open/close 1365 (time 1780336658.51 total 72.87 last 22.77) - open/close 1485 (time 1780336669.19 total 83.55 last 11.24) - open/close 1622 (time 1780336679.35 total 93.70 last 13.49) - open/close 1860 (time 1780336689.37 total 103.72 last 23.75) - open/close 2081 (time 1780336699.37 total 113.73 last 22.10) - open/close 2333 (time 1780336710.58 total 124.94 last 22.47) - open/close 2628 (time 1780336720.97 total 135.33 last 28.40) - open/close 2857 (time 1780336730.97 total 145.33 last 22.90) - open/close 3117 (time 1780336741.63 total 155.99 last 24.39) - open/close 3361 (time 1780336751.63 total 165.99 last 24.40) - open/close 3532 (time 1780336762.27 total 176.62 last 16.08) - open/close 3692 (time 1780336773.22 total 187.58 last 14.60) - open/close 3861 (time 1780336784.13 total 198.49 last 15.49) - open/close 3998 (time 1780336794.83 total 209.18 last 12.81) - open/close 4176 (time 1780336804.84 total 219.19 last 17.78) - open/close 4300 (time 1780336814.84 total 229.19 last 12.40) - open/close 4454 (time 1780336825.03 total 239.38 last 15.11) - open/close 4637 (time 1780336835.32 total 249.67 last 17.78) - open/close 4826 (time 1780336845.32 total 259.68 last 18.89) - open/close 5066 (time 1780336855.40 total 269.76 last 23.80) - open/close 5312 (time 1780336865.43 total 279.79 last 24.54) - open/close 5619 (time 1780336875.43 total 289.79 last 30.69) - open/close 5878 (time 1780336885.95 total 300.31 last 24.64) - open/close 6087 (time 1780336895.96 total 310.31 last 20.88) - open/close 6388 (time 1780336905.96 total 320.32 last 30.07) - open/close 6556 (time 1780336916.60 total 330.96 last 15.79) - open/close 6784 (time 1780336926.78 total 341.14 last 22.40) - open/close 7030 (time 1780336936.79 total 351.15 last 24.59) - open/close 7291 (time 1780336946.80 total 361.16 last 26.07) - open/close 7517 (time 1780336956.86 total 371.21 last 22.47) - open/close 7685 (time 1780336966.87 total 381.23 last 16.78) - open/close 7983 (time 1780336977.22 total 391.57 last 28.80) - open/close 8167 (time 1780336987.97 total 402.33 last 17.11) - open/close 8392 (time 1780336997.98 total 412.34 last 22.48) - open/close 8578 (time 1780337007.99 total 422.35 last 18.57) - open/close 8838 (time 1780337018.42 total 432.78 last 24.93) - open/close 9127 (time 1780337028.43 total 442.79 last 28.88) - open/close 9340 (time 1780337040.56 total 454.92 last 17.56) - open/close 9493 (time 1780337051.13 total 465.48 last 14.48) - open/close 9621 (time 1780337062.16 total 476.51 last 11.60) - open/close 9848 (time 1780337072.17 total 486.53 last 22.67) - open/close 9989 (time 1780337083.13 total 497.49 last 12.86) - open/close 10075 (time 1780337093.52 total 507.88 last 8.28) - open/close 10155 (time 1780337103.56 total 517.92 last 7.97) - open/close 10260 (time 1780337113.57 total 527.92 last 10.49) - open/close 10435 (time 1780337123.77 total 538.13 last 17.14) - open/close 10607 (time 1780337134.29 total 548.64 last 16.36) - open/close 10872 (time 1780337144.30 total 558.66 last 26.46) - open/close 11086 (time 1780337154.31 total 568.67 last 21.37) - open/close 11308 (time 1780337164.74 total 579.10 last 21.29) - open/close 11481 (time 1780337174.76 total 589.11 last 17.28) - open/close 11733 (time 1780337185.67 total 600.02 last 23.10) - open/close 12003 (time 1780337196.38 total 610.74 last 25.20) - open/close 12284 (time 1780337206.64 total 621.00 last 27.39) - open/close 12507 (time 1780337216.95 total 631.31 last 21.62) - open/close 12670 (time 1780337227.46 total 641.81 last 15.52) - open/close 12952 (time 1780337237.47 total 651.83 last 28.17) - open/close 13081 (time 1780337247.47 total 661.83 last 12.90) - open/close 13347 (time 1780337257.48 total 671.84 last 26.58) - open/close 13567 (time 1780337267.48 total 681.84 last 21.99) - open/close 13813 (time 1780337277.88 total 692.24 last 23.65) - open/close 14097 (time 1780337287.89 total 702.25 last 28.38) - open/close 14357 (time 1780337297.90 total 712.26 last 25.98) - open/close 14579 (time 1780337308.85 total 723.21 last 20.27) - open/close 14793 (time 1780337318.86 total 733.21 last 21.39) - open/close 15063 (time 1780337329.87 total 744.23 last 24.52) - open/close 15345 (time 1780337339.88 total 754.23 last 28.17) - open/close 15599 (time 1780337350.75 total 765.11 last 23.36) - open/close 15883 (time 1780337360.78 total 775.13 last 28.33) - open/close 16122 (time 1780337370.78 total 785.14 last 23.88) - open/close 16328 (time 1780337380.80 total 795.15 last 20.57) - open/close 16583 (time 1780337391.61 total 805.97 last 23.57) - open/close 16782 (time 1780337401.72 total 816.08 last 19.69) - open/close 17018 (time 1780337411.72 total 826.08 last 23.60) - open/close 17274 (time 1780337421.73 total 836.09 last 25.59) - open/close 17559 (time 1780337432.75 total 847.11 last 25.85) - open/close 17726 (time 1780337443.13 total 857.49 last 16.09) - open/close 17992 (time 1780337453.99 total 868.35 last 24.49) - open/close 18224 (time 1780337465.00 total 879.36 last 21.06) - open/close 18306 (time 1780337475.26 total 889.61 last 8.00) - open/close 18473 (time 1780337486.09 total 900.44 last 15.42) - open/close 18675 (time 1780337496.09 total 910.45 last 20.19) - open/close 18952 (time 1780337506.10 total 920.45 last 27.69) - open/close 19203 (time 1780337516.11 total 930.47 last 25.06) - open/close 19365 (time 1780337527.22 total 941.58 last 14.58) - open/close 19620 (time 1780337537.55 total 951.91 last 24.69) - open/close 19903 (time 1780337548.04 total 962.39 last 26.99) - open/close 20000 (time 1780337551.98 total 966.34 last 24.58) - open/close 20182 (time 1780337562.82 total 977.18 last 16.79) - open/close 20362 (time 1780337573.39 total 987.74 last 17.04) - open/close 20803 (time 1780337583.39 total 997.75 last 44.07) - open/close 21833 (time 1780337593.40 total 1007.75 last 102.96) - open/close 22925 (time 1780337603.40 total 1017.76 last 109.13) - open/close 23942 (time 1780337613.41 total 1027.76 last 101.67) - open/close 24933 (time 1780337623.41 total 1037.77 last 99.07) - open/close 25795 (time 1780337633.42 total 1047.78 last 86.10) - open/close 26705 (time 1780337643.43 total 1057.79 last 90.91) - open/close 27655 (time 1780337653.43 total 1067.79 last 94.95) - open/close 28671 (time 1780337663.44 total 1077.80 last 101.57) - open/close 29742 (time 1780337673.44 total 1087.80 last 107.04) - open/close 30000 (time 1780337675.73 total 1090.09 last 112.86) - open/close 30865 (time 1780337685.74 total 1100.09 last 86.45) - open/close 31835 (time 1780337695.74 total 1110.09 last 97.00) - open/close 32826 (time 1780337705.74 total 1120.10 last 99.06) - open/close 33881 (time 1780337715.74 total 1130.10 last 105.49) - open/close 34882 (time 1780337725.74 total 1140.10 last 100.09) - open/close 35751 (time 1780337735.75 total 1150.10 last 86.86) - open/close 36753 (time 1780337745.75 total 1160.11 last 100.15) - open/close 37568 (time 1780337755.76 total 1170.12 last 81.42) - open/close 38459 (time 1780337765.77 total 1180.12 last 89.04) - open/close 39425 (time 1780337775.77 total 1190.13 last 96.59) - open/close 40000 (time 1780337781.83 total 1196.19 last 94.78) - open/close 40935 (time 1780337791.84 total 1206.20 last 93.43) - open/close 41950 (time 1780337801.84 total 1216.20 last 101.48) - open/close 42865 (time 1780337811.86 total 1226.21 last 91.39) - open/close 43909 (time 1780337821.86 total 1236.21 last 104.39) - open/close 44718 (time 1780337831.86 total 1246.22 last 80.84) - open/close 45779 (time 1780337841.87 total 1256.23 last 106.03) - open/close 46684 (time 1780337851.88 total 1266.24 last 90.44) - open/close 47677 (time 1780337861.88 total 1276.24 last 99.25) - open/close 48625 (time 1780337871.89 total 1286.25 last 94.73) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1295.99 seconds: 38.30 ops/second - unlinked 0 (time 1780337885 ; total 0 ; last 0) - unlinked 10000 (time 1780338009 ; total 124 ; last 124) - unlinked 20000 (time 1780338123 ; total 238 ; last 114) - unlinked 30000 (time 1780338186 ; total 301 ; last 63) - unlinked 40000 (time 1780338249 ; total 364 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 429 seconds: 115.689980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2