== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:46:58 (1763340418) start mds service on oleg143-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 60 (time 1763340502.15 total 10.11 last 5.93) - open/close 201 (time 1763340512.18 total 20.15 last 14.05) - open/close 371 (time 1763340524.56 total 32.52 last 13.74) - open/close 546 (time 1763340535.41 total 43.37 last 16.13) - open/close 710 (time 1763340546.23 total 54.19 last 15.15) - open/close 840 (time 1763340556.24 total 64.20 last 12.99) - open/close 960 (time 1763340566.39 total 74.35 last 11.82) - open/close 1119 (time 1763340577.03 total 84.99 last 14.94) - open/close 1238 (time 1763340587.89 total 95.85 last 10.95) - open/close 1372 (time 1763340598.00 total 105.96 last 13.25) - open/close 1536 (time 1763340608.25 total 116.21 last 16.01) - open/close 1688 (time 1763340618.43 total 126.39 last 14.93) - open/close 1818 (time 1763340628.54 total 136.50 last 12.86) - open/close 1952 (time 1763340638.79 total 146.76 last 13.07) - open/close 2239 (time 1763340649.58 total 157.55 last 26.60) - open/close 2419 (time 1763340659.65 total 167.61 last 17.88) - open/close 2611 (time 1763340670.65 total 178.61 last 17.46) - open/close 2805 (time 1763340681.65 total 189.62 last 17.63) - open/close 2938 (time 1763340691.66 total 199.62 last 13.30) - open/close 3127 (time 1763340702.59 total 210.55 last 17.28) - open/close 3297 (time 1763340712.60 total 220.56 last 16.99) - open/close 3498 (time 1763340722.93 total 230.89 last 19.45) - open/close 3710 (time 1763340733.12 total 241.08 last 20.81) - open/close 3914 (time 1763340743.74 total 251.70 last 19.21) - open/close 4141 (time 1763340753.84 total 261.80 last 22.46) - open/close 4324 (time 1763340763.84 total 271.81 last 18.30) - open/close 4494 (time 1763340773.86 total 281.82 last 16.97) - open/close 4714 (time 1763340783.93 total 291.89 last 21.85) - open/close 4956 (time 1763340794.74 total 302.70 last 22.38) - open/close 5189 (time 1763340804.74 total 312.70 last 23.30) - open/close 5386 (time 1763340814.84 total 322.80 last 19.51) - open/close 5556 (time 1763340824.85 total 332.81 last 16.98) - open/close 5754 (time 1763340834.88 total 342.84 last 19.75) - open/close 5935 (time 1763340844.88 total 352.84 last 18.09) - open/close 6163 (time 1763340854.90 total 362.86 last 22.75) - open/close 6369 (time 1763340864.93 total 372.89 last 20.54) - open/close 6523 (time 1763340874.94 total 382.90 last 15.38) - open/close 6674 (time 1763340885.43 total 393.39 last 14.40) - open/close 6803 (time 1763340895.54 total 403.50 last 12.76) - open/close 6949 (time 1763340905.97 total 413.93 last 13.99) - open/close 7186 (time 1763340916.86 total 424.82 last 21.77) - open/close 7322 (time 1763340927.61 total 435.57 last 12.65) - open/close 7523 (time 1763340937.63 total 445.59 last 20.06) - open/close 7695 (time 1763340947.77 total 455.73 last 16.96) - open/close 7941 (time 1763340958.13 total 466.09 last 23.74) - open/close 8098 (time 1763340968.69 total 476.66 last 14.86) - open/close 8221 (time 1763340978.70 total 486.66 last 12.29) - open/close 8421 (time 1763340988.74 total 496.70 last 19.93) - open/close 8604 (time 1763340999.67 total 507.63 last 16.74) - open/close 8814 (time 1763341009.98 total 517.94 last 20.36) - open/close 8982 (time 1763341020.86 total 528.83 last 15.43) - open/close 9213 (time 1763341031.22 total 539.18 last 22.30) - open/close 9447 (time 1763341041.23 total 549.19 last 23.39) - open/close 9673 (time 1763341051.25 total 559.22 last 22.54) - open/close 9898 (time 1763341062.01 total 569.97 last 20.92) - open/close 10000 (time 1763341066.90 total 574.87 last 20.84) - open/close 10168 (time 1763341076.92 total 584.88 last 16.78) - open/close 10340 (time 1763341086.92 total 594.89 last 17.19) - open/close 10569 (time 1763341096.94 total 604.90 last 22.87) - open/close 10775 (time 1763341107.14 total 615.11 last 20.18) - open/close 10979 (time 1763341117.44 total 625.40 last 19.82) - open/close 11258 (time 1763341129.60 total 637.57 last 22.93) - open/close 11463 (time 1763341140.55 total 648.51 last 18.73) - open/close 11702 (time 1763341150.58 total 658.54 last 23.84) - open/close 11890 (time 1763341160.58 total 668.54 last 18.79) - open/close 12119 (time 1763341171.01 total 678.97 last 21.96) - open/close 12338 (time 1763341181.25 total 689.22 last 21.37) - open/close 12553 (time 1763341191.26 total 699.22 last 21.48) - open/close 12784 (time 1763341201.26 total 709.23 last 23.10) - open/close 13016 (time 1763341211.27 total 719.23 last 23.19) - open/close 13705 (time 1763341221.28 total 729.24 last 68.84) - open/close 14494 (time 1763341231.28 total 739.24 last 78.89) - open/close 15230 (time 1763341241.30 total 749.26 last 73.49) - open/close 15974 (time 1763341251.30 total 759.26 last 74.40) - open/close 16732 (time 1763341261.31 total 769.27 last 75.72) - open/close 17418 (time 1763341271.31 total 779.27 last 68.58) - open/close 18163 (time 1763341281.32 total 789.28 last 74.40) - open/close 18855 (time 1763341291.34 total 799.30 last 69.08) - open/close 19546 (time 1763341301.35 total 809.32 last 69.00) - open/close 20000 (time 1763341307.59 total 815.55 last 72.77) - open/close 20674 (time 1763341317.60 total 825.56 last 67.38) - open/close 21423 (time 1763341327.60 total 835.56 last 74.89) - open/close 22200 (time 1763341337.60 total 845.57 last 77.65) - open/close 22919 (time 1763341347.61 total 855.57 last 71.84) - open/close 23678 (time 1763341357.62 total 865.58 last 75.84) - open/close 24326 (time 1763341367.63 total 875.59 last 64.72) - open/close 25034 (time 1763341377.64 total 885.60 last 70.77) - open/close 25726 (time 1763341387.64 total 895.61 last 69.15) - open/close 26453 (time 1763341397.65 total 905.61 last 72.65) - open/close 27225 (time 1763341407.66 total 915.62 last 77.16) - open/close 28003 (time 1763341417.67 total 925.64 last 77.67) - open/close 28733 (time 1763341427.68 total 935.64 last 72.94) - open/close 29417 (time 1763341437.69 total 945.65 last 68.33) - open/close 30000 (time 1763341446.97 total 954.93 last 62.85) - open/close 30566 (time 1763341456.97 total 964.93 last 56.58) - open/close 31094 (time 1763341466.98 total 974.94 last 52.76) - open/close 31737 (time 1763341476.99 total 984.95 last 64.25) - open/close 32479 (time 1763341487.00 total 994.96 last 74.11) - open/close 33239 (time 1763341497.01 total 1004.97 last 75.89) - open/close 33946 (time 1763341507.02 total 1014.98 last 70.63) - open/close 34618 (time 1763341517.03 total 1025.00 last 67.12) - open/close 35364 (time 1763341527.04 total 1035.01 last 74.53) - open/close 36086 (time 1763341537.05 total 1045.01 last 72.14) - open/close 36773 (time 1763341547.06 total 1055.02 last 68.66) - open/close 37445 (time 1763341557.07 total 1065.03 last 67.14) - open/close 38183 (time 1763341567.07 total 1075.03 last 73.80) - open/close 38956 (time 1763341577.07 total 1085.03 last 77.29) - open/close 39681 (time 1763341587.08 total 1095.05 last 72.37) - open/close 40000 (time 1763341591.71 total 1099.67 last 69.01) - open/close 40652 (time 1763341601.71 total 1109.68 last 65.16) - open/close 41325 (time 1763341611.72 total 1119.68 last 67.27) - open/close 42053 (time 1763341621.72 total 1129.68 last 72.78) - open/close 42796 (time 1763341631.75 total 1139.71 last 74.08) - open/close 43475 (time 1763341641.76 total 1149.72 last 67.82) - open/close 44137 (time 1763341651.77 total 1159.73 last 66.14) - open/close 44892 (time 1763341661.77 total 1169.74 last 75.48) - open/close 45676 (time 1763341671.78 total 1179.74 last 78.36) - open/close 46401 (time 1763341681.78 total 1189.75 last 72.46) - open/close 47121 (time 1763341691.80 total 1199.76 last 71.91) - open/close 47805 (time 1763341701.81 total 1209.77 last 68.35) - open/close 48540 (time 1763341711.81 total 1219.77 last 73.50) - open/close 49274 (time 1763341721.82 total 1229.78 last 73.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1234.85 seconds: 40.19 ops/second - unlinked 0 (time 1763341730 ; total 0 ; last 0) - unlinked 10000 (time 1763341860 ; total 130 ; last 130) - unlinked 20000 (time 1763341995 ; total 265 ; last 135) - unlinked 30000 (time 1763342129 ; total 399 ; last 134) - unlinked 40000 (time 1763342271 ; total 541 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 680 seconds: 72.988235 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg143-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2