== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:24:30 (1789125870) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1789125960.27 total 10.00 last 22.09) - open/close 485 (time 1789125970.27 total 20.01 last 26.39) - open/close 725 (time 1789125981.94 total 31.67 last 20.57) - open/close 852 (time 1789125992.62 total 42.35 last 11.89) - open/close 1051 (time 1789126003.44 total 53.18 last 18.38) - open/close 1207 (time 1789126016.05 total 65.79 last 12.37) - open/close 1449 (time 1789126026.06 total 75.79 last 24.19) - open/close 1715 (time 1789126036.52 total 86.25 last 25.43) - open/close 1943 (time 1789126046.53 total 96.26 last 22.78) - open/close 2161 (time 1789126058.70 total 108.43 last 17.91) - open/close 2395 (time 1789126068.70 total 118.44 last 23.38) - open/close 2628 (time 1789126079.60 total 129.33 last 21.39) - open/close 2838 (time 1789126089.63 total 139.37 last 20.93) - open/close 3073 (time 1789126099.64 total 149.38 last 23.47) - open/close 3237 (time 1789126110.06 total 159.80 last 15.74) - open/close 3345 (time 1789126120.58 total 170.32 last 10.27) - open/close 3608 (time 1789126131.17 total 180.91 last 24.83) - open/close 3832 (time 1789126141.17 total 190.91 last 22.40) - open/close 4057 (time 1789126151.41 total 201.14 last 21.99) - open/close 4270 (time 1789126162.47 total 212.20 last 19.25) - open/close 4467 (time 1789126172.49 total 222.22 last 19.67) - open/close 4729 (time 1789126182.96 total 232.69 last 25.01) - open/close 4914 (time 1789126193.31 total 243.05 last 17.87) - open/close 5177 (time 1789126205.28 total 255.02 last 21.97) - open/close 5419 (time 1789126215.29 total 265.02 last 24.20) - open/close 5632 (time 1789126225.65 total 275.38 last 20.56) - open/close 5891 (time 1789126236.60 total 286.33 last 23.65) - open/close 6132 (time 1789126247.27 total 297.00 last 22.59) - open/close 6364 (time 1789126257.28 total 307.01 last 23.17) - open/close 6607 (time 1789126267.28 total 317.01 last 24.30) - open/close 6810 (time 1789126277.29 total 327.02 last 20.28) - open/close 7047 (time 1789126287.60 total 337.33 last 22.99) - open/close 7276 (time 1789126297.66 total 347.40 last 22.75) - open/close 7485 (time 1789126307.88 total 357.62 last 20.45) - open/close 7621 (time 1789126318.06 total 367.80 last 13.36) - open/close 7796 (time 1789126328.49 total 378.22 last 16.78) - open/close 8039 (time 1789126339.44 total 389.17 last 22.20) - open/close 8248 (time 1789126349.55 total 399.28 last 20.67) - open/close 8448 (time 1789126359.57 total 409.30 last 19.97) - open/close 8669 (time 1789126369.58 total 419.31 last 22.08) - open/close 8891 (time 1789126379.75 total 429.49 last 21.82) - open/close 9084 (time 1789126391.60 total 441.33 last 16.29) - open/close 9324 (time 1789126401.61 total 451.34 last 23.99) - open/close 9531 (time 1789126411.61 total 461.34 last 20.69) - open/close 9650 (time 1789126422.27 total 472.01 last 11.16) - open/close 9809 (time 1789126432.81 total 482.55 last 15.09) - open/close 10000 (time 1789126438.81 total 488.54 last 31.85) - open/close 10230 (time 1789126448.82 total 498.55 last 22.98) - open/close 10385 (time 1789126459.75 total 509.49 last 14.17) - open/close 10616 (time 1789126469.76 total 519.50 last 23.08) - open/close 10736 (time 1789126479.78 total 529.51 last 11.98) - open/close 10974 (time 1789126491.73 total 541.47 last 19.91) - open/close 11103 (time 1789126502.56 total 552.30 last 11.91) - open/close 11367 (time 1789126512.57 total 562.30 last 26.38) - open/close 11513 (time 1789126523.94 total 573.67 last 12.84) - open/close 11647 (time 1789126534.33 total 584.07 last 12.89) - open/close 11889 (time 1789126544.88 total 594.61 last 22.94) - open/close 12176 (time 1789126554.89 total 604.62 last 28.68) - open/close 12377 (time 1789126565.99 total 615.72 last 18.11) - open/close 12521 (time 1789126576.00 total 625.74 last 14.37) - open/close 12652 (time 1789126586.02 total 635.75 last 13.08) - open/close 12881 (time 1789126596.02 total 645.76 last 22.89) - open/close 12981 (time 1789126606.04 total 655.78 last 9.98) - open/close 13185 (time 1789126616.68 total 666.42 last 19.17) - open/close 13479 (time 1789126626.70 total 676.44 last 29.34) - open/close 13661 (time 1789126637.42 total 687.16 last 16.97) - open/close 13853 (time 1789126647.43 total 697.16 last 19.19) - open/close 13925 (time 1789126657.44 total 707.17 last 7.19) - open/close 14069 (time 1789126667.51 total 717.25 last 14.29) - open/close 14166 (time 1789126680.97 total 730.70 last 7.21) - open/close 14438 (time 1789126690.97 total 740.70 last 27.19) - open/close 14679 (time 1789126701.44 total 751.18 last 23.01) - open/close 14932 (time 1789126711.47 total 761.20 last 25.23) - open/close 15204 (time 1789126721.47 total 771.20 last 27.20) - open/close 15417 (time 1789126733.37 total 783.10 last 17.90) - open/close 15669 (time 1789126743.38 total 793.11 last 25.17) - open/close 15785 (time 1789126754.09 total 803.82 last 10.83) - open/close 15904 (time 1789126764.10 total 813.83 last 11.89) - open/close 16072 (time 1789126774.80 total 824.53 last 15.71) - open/close 16318 (time 1789126784.81 total 834.54 last 24.57) - open/close 16587 (time 1789126795.24 total 844.97 last 25.79) - open/close 16802 (time 1789126805.55 total 855.29 last 20.84) - open/close 17048 (time 1789126816.10 total 865.84 last 23.32) - open/close 17287 (time 1789126826.18 total 875.91 last 23.73) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17425) error: No space left on device total: 17425 open/close in 880.42 seconds: 19.79 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76069) - No such process - unlinked 0 (time 1789126843 ; total 0 ; last 0) - unlinked 10000 (time 1789126962 ; total 119 ; last 119) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17425) error: No such file or directory total: 17425 unlinks in 183 seconds: 95.218582 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local oleg111-server: modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2