== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:45:50 (1783140350) start mds service on oleg148-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg148-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg148-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 256 (time 1783140456.91 total 10.01 last 25.59) - open/close 510 (time 1783140467.64 total 20.73 last 23.68) - open/close 734 (time 1783140478.45 total 31.54 last 20.72) - open/close 954 (time 1783140488.45 total 41.55 last 21.98) - open/close 1190 (time 1783140498.47 total 51.57 last 23.55) - open/close 1431 (time 1783140509.29 total 62.39 last 22.28) - open/close 1695 (time 1783140519.34 total 72.44 last 26.27) - open/close 1931 (time 1783140529.35 total 82.45 last 23.58) - open/close 2175 (time 1783140539.36 total 92.45 last 24.38) - open/close 2420 (time 1783140550.07 total 103.17 last 22.87) - open/close 2625 (time 1783140561.07 total 114.17 last 18.64) - open/close 2875 (time 1783140571.84 total 124.93 last 23.22) - open/close 3066 (time 1783140581.85 total 134.94 last 19.08) - open/close 3293 (time 1783140592.63 total 145.73 last 21.05) - open/close 3492 (time 1783140603.56 total 156.66 last 18.21) - open/close 3785 (time 1783140614.20 total 167.30 last 27.53) - open/close 4038 (time 1783140624.37 total 177.47 last 24.88) - open/close 4282 (time 1783140634.38 total 187.47 last 24.39) - open/close 4473 (time 1783140644.39 total 197.49 last 19.07) - open/close 4731 (time 1783140654.65 total 207.74 last 25.15) - open/close 5007 (time 1783140665.21 total 218.30 last 26.13) - open/close 5274 (time 1783140675.22 total 228.31 last 26.68) - open/close 5433 (time 1783140686.18 total 239.28 last 14.50) - open/close 5672 (time 1783140696.44 total 249.54 last 23.30) - open/close 5843 (time 1783140706.50 total 259.60 last 17.00) - open/close 6074 (time 1783140716.66 total 269.75 last 22.75) - open/close 6308 (time 1783140727.37 total 280.47 last 21.84) - open/close 6452 (time 1783140737.77 total 290.87 last 13.84) - open/close 6619 (time 1783140748.43 total 301.52 last 15.67) - open/close 6868 (time 1783140759.08 total 312.18 last 23.38) - open/close 7076 (time 1783140769.22 total 322.32 last 20.51) - open/close 7348 (time 1783140779.22 total 332.32 last 27.20) - open/close 7577 (time 1783140789.55 total 342.65 last 22.17) - open/close 7845 (time 1783140799.55 total 352.65 last 26.80) - open/close 8055 (time 1783140809.64 total 362.74 last 20.82) - open/close 8235 (time 1783140820.28 total 373.37 last 16.92) - open/close 8499 (time 1783140831.22 total 384.32 last 24.13) - open/close 8687 (time 1783140842.04 total 395.13 last 17.38) - open/close 8975 (time 1783140852.48 total 405.58 last 27.58) - open/close 9236 (time 1783140863.02 total 416.11 last 24.77) - open/close 9433 (time 1783140874.03 total 427.13 last 17.88) - open/close 9640 (time 1783140884.08 total 437.17 last 20.60) - open/close 9777 (time 1783140894.08 total 447.18 last 13.70) - open/close 10000 (time 1783140902.48 total 455.58 last 26.56) - open/close 10294 (time 1783140912.48 total 465.58 last 29.39) - open/close 10486 (time 1783140922.50 total 475.60 last 19.16) - open/close 10698 (time 1783140932.51 total 485.60 last 21.19) - open/close 10855 (time 1783140942.90 total 495.99 last 15.11) - open/close 11135 (time 1783140952.90 total 505.99 last 28.00) - open/close 11341 (time 1783140963.26 total 516.36 last 19.88) - open/close 11524 (time 1783140973.26 total 526.36 last 18.29) - open/close 11727 (time 1783140983.27 total 536.36 last 20.29) - open/close 11819 (time 1783140994.03 total 547.12 last 8.55) - open/close 12043 (time 1783141004.39 total 557.49 last 21.61) - open/close 12169 (time 1783141014.40 total 567.49 last 12.60) - open/close 12303 (time 1783141025.33 total 578.42 last 12.26) - open/close 12527 (time 1783141035.33 total 588.43 last 22.39) - open/close 12736 (time 1783141046.26 total 599.35 last 19.14) - open/close 12856 (time 1783141056.36 total 609.46 last 11.88) - open/close 13077 (time 1783141066.36 total 619.46 last 22.09) - open/close 13230 (time 1783141076.79 total 629.88 last 14.68) - open/close 13407 (time 1783141086.85 total 639.94 last 17.60) - open/close 13568 (time 1783141097.33 total 650.42 last 15.36) - open/close 13742 (time 1783141107.69 total 660.79 last 16.78) - open/close 13894 (time 1783141117.95 total 671.04 last 14.82) - open/close 14081 (time 1783141128.19 total 681.28 last 18.26) - open/close 14399 (time 1783141138.79 total 691.89 last 29.98) - open/close 14948 (time 1783141148.80 total 701.89 last 54.88) - open/close 15905 (time 1783141158.80 total 711.90 last 95.68) - open/close 17089 (time 1783141168.80 total 721.90 last 118.37) - open/close 18267 (time 1783141178.81 total 731.90 last 117.73) - open/close 19302 (time 1783141188.81 total 741.90 last 103.50) - open/close 20000 (time 1783141195.81 total 748.91 last 99.62) - open/close 20953 (time 1783141205.82 total 758.91 last 95.26) - open/close 21861 (time 1783141215.82 total 768.92 last 90.79) - open/close 22840 (time 1783141225.82 total 778.92 last 97.85) - open/close 23853 (time 1783141235.83 total 788.92 last 101.29) - open/close 25024 (time 1783141245.83 total 798.92 last 117.06) - open/close 25994 (time 1783141255.83 total 808.93 last 96.99) - open/close 27070 (time 1783141265.83 total 818.93 last 107.59) - open/close 28197 (time 1783141275.83 total 828.93 last 112.68) - open/close 29318 (time 1783141285.83 total 838.93 last 112.09) - open/close 30000 (time 1783141291.88 total 844.98 last 112.75) - open/close 31124 (time 1783141301.89 total 854.99 last 112.29) - open/close 32244 (time 1783141311.90 total 864.99 last 111.96) - open/close 33357 (time 1783141321.90 total 874.99 last 111.28) - open/close 34465 (time 1783141331.90 total 885.00 last 110.78) - open/close 35651 (time 1783141341.91 total 895.00 last 118.52) - open/close 36799 (time 1783141351.91 total 905.01 last 114.74) - open/close 38008 (time 1783141361.91 total 915.01 last 120.87) - open/close 39091 (time 1783141371.91 total 925.01 last 108.30) - open/close 40000 (time 1783141380.40 total 933.50 last 107.12) - open/close 41103 (time 1783141390.40 total 943.50 last 110.28) - open/close 42269 (time 1783141400.40 total 953.50 last 116.57) - open/close 43390 (time 1783141410.41 total 963.51 last 112.02) - open/close 44505 (time 1783141420.41 total 973.51 last 111.46) - open/close 45684 (time 1783141430.43 total 983.53 last 117.68) - open/close 46840 (time 1783141440.44 total 993.53 last 115.57) - open/close 47943 (time 1783141450.44 total 1003.54 last 110.22) - open/close 49081 (time 1783141460.47 total 1013.57 last 113.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1019.77 seconds: 48.67 ops/second - unlinked 0 (time 1783141472 ; total 0 ; last 0) - unlinked 10000 (time 1783141634 ; total 162 ; last 162) - unlinked 20000 (time 1783141723 ; total 251 ; last 89) - unlinked 30000 (time 1783141795 ; total 323 ; last 72) - unlinked 40000 (time 1783141867 ; total 395 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 464 seconds: 106.963364 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: modules unloaded. pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2