== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:10:08 (1783141808) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg346-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg346-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 148 (time 1783141907.40 total 10.01 last 14.79) - open/close 431 (time 1783141918.18 total 20.79 last 26.24) - open/close 649 (time 1783141929.09 total 31.70 last 19.98) - open/close 905 (time 1783141939.10 total 41.71 last 25.58) - open/close 1138 (time 1783141949.35 total 51.96 last 22.74) - open/close 1385 (time 1783141959.36 total 61.97 last 24.67) - open/close 1645 (time 1783141969.45 total 72.06 last 25.78) - open/close 1905 (time 1783141979.63 total 82.24 last 25.54) - open/close 2164 (time 1783141990.12 total 92.73 last 24.68) - open/close 2395 (time 1783142001.07 total 103.68 last 21.10) - open/close 2637 (time 1783142011.29 total 113.90 last 23.67) - open/close 2861 (time 1783142021.31 total 123.92 last 22.36) - open/close 3069 (time 1783142032.23 total 134.84 last 19.06) - open/close 3329 (time 1783142042.87 total 145.48 last 24.43) - open/close 3432 (time 1783142053.43 total 156.04 last 9.75) - open/close 3667 (time 1783142063.48 total 166.09 last 23.38) - open/close 3935 (time 1783142073.58 total 176.19 last 26.54) - open/close 4180 (time 1783142084.21 total 186.82 last 23.06) - open/close 4431 (time 1783142094.62 total 197.23 last 24.10) - open/close 4674 (time 1783142104.82 total 207.43 last 23.83) - open/close 4940 (time 1783142115.70 total 218.31 last 24.45) - open/close 5198 (time 1783142125.71 total 228.32 last 25.77) - open/close 5484 (time 1783142136.75 total 239.36 last 25.90) - open/close 5760 (time 1783142146.80 total 249.41 last 27.47) - open/close 6007 (time 1783142157.16 total 259.76 last 23.86) - open/close 6261 (time 1783142167.24 total 269.85 last 25.19) - open/close 6491 (time 1783142177.26 total 279.87 last 22.95) - open/close 6762 (time 1783142187.27 total 289.88 last 27.07) - open/close 6944 (time 1783142197.67 total 300.28 last 17.50) - open/close 7234 (time 1783142207.67 total 310.28 last 29.00) - open/close 7477 (time 1783142218.67 total 321.28 last 22.09) - open/close 7706 (time 1783142229.42 total 332.03 last 21.31) - open/close 7900 (time 1783142240.32 total 342.93 last 17.80) - open/close 8136 (time 1783142250.81 total 353.42 last 22.50) - open/close 8372 (time 1783142261.36 total 363.97 last 22.35) - open/close 8598 (time 1783142273.77 total 376.38 last 18.22) - open/close 8845 (time 1783142284.33 total 386.94 last 23.38) - open/close 9120 (time 1783142295.36 total 397.97 last 24.93) - open/close 9416 (time 1783142305.38 total 407.99 last 29.54) - open/close 9647 (time 1783142315.43 total 418.04 last 22.99) - open/close 9880 (time 1783142325.45 total 428.05 last 23.27) - open/close 10000 (time 1783142330.65 total 433.26 last 23.04) - open/close 10240 (time 1783142340.77 total 443.38 last 23.72) - open/close 10476 (time 1783142351.15 total 453.75 last 22.75) - open/close 10564 (time 1783142361.20 total 463.81 last 8.75) - open/close 10827 (time 1783142372.14 total 474.74 last 24.05) - open/close 11074 (time 1783142382.25 total 484.86 last 24.42) - open/close 11306 (time 1783142393.05 total 495.66 last 21.48) - open/close 11543 (time 1783142403.30 total 505.91 last 23.12) - open/close 11788 (time 1783142413.48 total 516.09 last 24.07) - open/close 12048 (time 1783142424.18 total 526.79 last 24.31) - open/close 12279 (time 1783142434.18 total 536.79 last 23.08) - open/close 12532 (time 1783142444.39 total 547.00 last 24.79) - open/close 12773 (time 1783142455.16 total 557.77 last 22.38) - open/close 12971 (time 1783142465.16 total 567.77 last 19.79) - open/close 13232 (time 1783142475.17 total 577.77 last 26.09) - open/close 13485 (time 1783142485.17 total 587.78 last 25.30) - open/close 13733 (time 1783142495.30 total 597.90 last 24.49) - open/close 13983 (time 1783142505.33 total 607.94 last 24.91) - open/close 14253 (time 1783142515.63 total 618.24 last 26.22) - open/close 14508 (time 1783142526.19 total 628.80 last 24.14) - open/close 14772 (time 1783142536.19 total 638.80 last 26.40) - open/close 15022 (time 1783142546.34 total 648.95 last 24.64) - open/close 15289 (time 1783142556.90 total 659.51 last 25.28) - open/close 15520 (time 1783142567.59 total 670.19 last 21.62) - open/close 15764 (time 1783142577.70 total 680.31 last 24.13) - open/close 15997 (time 1783142587.70 total 690.31 last 23.28) - open/close 16223 (time 1783142598.19 total 700.80 last 21.55) - open/close 16455 (time 1783142608.68 total 711.29 last 22.12) - open/close 16716 (time 1783142619.57 total 722.18 last 23.98) - open/close 16974 (time 1783142629.57 total 732.18 last 25.79) - open/close 17195 (time 1783142639.97 total 742.58 last 21.24) - open/close 17323 (time 1783142649.99 total 752.59 last 12.78) - open/close 17480 (time 1783142660.20 total 762.81 last 15.37) - open/close 17703 (time 1783142670.21 total 772.82 last 22.27) - open/close 17950 (time 1783142680.61 total 783.22 last 23.76) - open/close 18141 (time 1783142691.44 total 794.05 last 17.63) - open/close 18407 (time 1783142702.24 total 804.85 last 24.62) - open/close 18639 (time 1783142712.49 total 815.10 last 22.63) - open/close 18829 (time 1783142722.50 total 825.11 last 18.99) - open/close 19036 (time 1783142732.51 total 835.12 last 20.68) - open/close 19393 (time 1783142742.52 total 845.13 last 35.67) - open/close 20000 (time 1783142749.70 total 852.31 last 84.51) - open/close 20845 (time 1783142759.71 total 862.31 last 84.45) - open/close 21709 (time 1783142769.71 total 872.31 last 86.40) - open/close 22515 (time 1783142779.71 total 882.32 last 80.57) - open/close 23347 (time 1783142789.71 total 892.32 last 83.16) - open/close 24224 (time 1783142799.71 total 902.32 last 87.69) - open/close 25186 (time 1783142809.72 total 912.33 last 96.13) - open/close 26052 (time 1783142819.73 total 922.34 last 86.53) - open/close 26900 (time 1783142829.73 total 932.34 last 84.78) - open/close 27775 (time 1783142839.74 total 942.35 last 87.44) - open/close 28701 (time 1783142849.76 total 952.36 last 92.45) - open/close 29573 (time 1783142859.77 total 962.38 last 87.10) - open/close 30000 (time 1783142864.70 total 967.31 last 86.51) - open/close 30929 (time 1783142874.71 total 977.32 last 92.87) - open/close 31920 (time 1783142884.71 total 987.32 last 99.05) - open/close 32915 (time 1783142894.72 total 997.33 last 99.41) - open/close 33928 (time 1783142904.73 total 1007.34 last 101.21) - open/close 34940 (time 1783142914.74 total 1017.35 last 101.10) - open/close 35881 (time 1783142924.75 total 1027.36 last 94.01) - open/close 37017 (time 1783142934.76 total 1037.37 last 113.49) - open/close 38067 (time 1783142944.76 total 1047.37 last 104.97) - open/close 39067 (time 1783142954.77 total 1057.37 last 99.95) - open/close 39965 (time 1783142964.78 total 1067.38 last 89.72) - open/close 40940 (time 1783142974.78 total 1077.39 last 97.45) - open/close 41944 (time 1783142984.78 total 1087.39 last 100.37) - open/close 42963 (time 1783142994.79 total 1097.40 last 101.86) - open/close 43995 (time 1783143004.79 total 1107.40 last 103.19) - open/close 45250 (time 1783143014.80 total 1117.40 last 125.41) - open/close 46355 (time 1783143024.80 total 1127.41 last 110.44) - open/close 47401 (time 1783143034.81 total 1137.41 last 104.54) - open/close 48528 (time 1783143044.81 total 1147.42 last 112.68) - open/close 49562 (time 1783143054.82 total 1157.43 last 103.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1158.16 seconds: 42.85 ops/second - unlinked 0 (time 1783143058 ; total 0 ; last 0) - unlinked 10000 (time 1783143138 ; total 80 ; last 80) - unlinked 20000 (time 1783143203 ; total 145 ; last 65) - unlinked 30000 (time 1783143242 ; total 184 ; last 39) - unlinked 40000 (time 1783143279 ; total 221 ; last 37) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 255 seconds: 194.631378 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2