== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:55:45 (1763337345) start mds service on oleg320-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions oleg320-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg320-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 226 (time 1763337418.24 total 10.01 last 22.58) - open/close 458 (time 1763337428.99 total 20.76 last 21.58) - open/close 712 (time 1763337439.00 total 30.76 last 25.38) - open/close 940 (time 1763337449.72 total 41.49 last 21.26) - open/close 1161 (time 1763337459.91 total 51.67 last 21.70) - open/close 1384 (time 1763337470.00 total 61.77 last 22.08) - open/close 1603 (time 1763337480.01 total 71.78 last 21.89) - open/close 1830 (time 1763337490.02 total 81.78 last 22.68) - open/close 2026 (time 1763337500.02 total 91.78 last 19.60) - open/close 2212 (time 1763337510.51 total 102.27 last 17.73) - open/close 2406 (time 1763337521.20 total 112.96 last 18.15) - open/close 2617 (time 1763337531.29 total 123.05 last 20.91) - open/close 2877 (time 1763337541.61 total 133.38 last 25.19) - open/close 3096 (time 1763337551.61 total 143.38 last 21.89) - open/close 3287 (time 1763337562.62 total 154.39 last 17.35) - open/close 3488 (time 1763337573.25 total 165.01 last 18.92) - open/close 3670 (time 1763337584.18 total 175.95 last 16.64) - open/close 3904 (time 1763337594.37 total 186.14 last 22.97) - open/close 4081 (time 1763337604.37 total 196.14 last 17.69) - open/close 4274 (time 1763337614.77 total 206.53 last 18.57) - open/close 4453 (time 1763337624.77 total 216.54 last 17.90) - open/close 4670 (time 1763337635.38 total 227.15 last 20.45) - open/close 4893 (time 1763337645.44 total 237.21 last 22.16) - open/close 5096 (time 1763337655.45 total 247.21 last 20.29) - open/close 5286 (time 1763337665.46 total 257.23 last 18.98) - open/close 5509 (time 1763337675.47 total 267.24 last 22.28) - open/close 5742 (time 1763337685.58 total 277.35 last 23.04) - open/close 5937 (time 1763337698.24 total 290.00 last 15.41) - open/close 6171 (time 1763337708.66 total 300.42 last 22.46) - open/close 6394 (time 1763337719.31 total 311.07 last 20.94) - open/close 6609 (time 1763337729.32 total 321.09 last 21.47) - open/close 6819 (time 1763337740.15 total 331.92 last 19.40) - open/close 6999 (time 1763337750.46 total 342.22 last 17.46) - open/close 7246 (time 1763337761.20 total 352.97 last 22.99) - open/close 7453 (time 1763337772.14 total 363.91 last 18.92) - open/close 7668 (time 1763337782.74 total 374.50 last 20.29) - open/close 7920 (time 1763337792.76 total 384.53 last 25.14) - open/close 8117 (time 1763337803.51 total 395.27 last 18.33) - open/close 8318 (time 1763337813.51 total 405.28 last 20.09) - open/close 8536 (time 1763337823.52 total 415.29 last 21.78) - open/close 8765 (time 1763337833.81 total 425.58 last 22.26) - open/close 8987 (time 1763337844.36 total 436.13 last 21.04) - open/close 9184 (time 1763337854.38 total 446.15 last 19.66) - open/close 9395 (time 1763337864.44 total 456.21 last 20.99) - open/close 9621 (time 1763337874.45 total 466.22 last 22.57) - open/close 9823 (time 1763337884.46 total 476.23 last 20.18) - open/close 10000 (time 1763337892.83 total 484.60 last 21.14) - open/close 10239 (time 1763337903.35 total 495.12 last 22.73) - open/close 10477 (time 1763337914.31 total 506.07 last 21.72) - open/close 10664 (time 1763337924.32 total 516.09 last 18.68) - open/close 10774 (time 1763337935.18 total 526.95 last 10.13) - open/close 10969 (time 1763337945.41 total 537.18 last 19.06) - open/close 11174 (time 1763337955.77 total 547.54 last 19.78) - open/close 11390 (time 1763337965.84 total 557.61 last 21.45) - open/close 11601 (time 1763337975.86 total 567.63 last 21.07) - open/close 11812 (time 1763337985.88 total 577.65 last 21.05) - open/close 12042 (time 1763337996.53 total 588.30 last 21.60) - open/close 12267 (time 1763338006.74 total 598.51 last 22.04) - open/close 12470 (time 1763338016.74 total 608.51 last 20.29) - open/close 12683 (time 1763338026.76 total 618.52 last 21.27) - open/close 12884 (time 1763338036.76 total 628.53 last 20.09) - open/close 13089 (time 1763338047.82 total 639.59 last 18.53) - open/close 13322 (time 1763338058.31 total 650.08 last 22.22) - open/close 13561 (time 1763338068.98 total 660.75 last 22.40) - open/close 13808 (time 1763338079.43 total 671.20 last 23.64) - open/close 14046 (time 1763338089.97 total 681.74 last 22.58) - open/close 14271 (time 1763338100.82 total 692.59 last 20.74) - open/close 14514 (time 1763338111.22 total 702.98 last 23.37) - open/close 14712 (time 1763338121.58 total 713.35 last 19.10) - open/close 14901 (time 1763338131.59 total 723.36 last 18.88) - open/close 15094 (time 1763338142.48 total 734.24 last 17.73) - open/close 15282 (time 1763338152.77 total 744.54 last 18.26) - open/close 15488 (time 1763338163.12 total 754.88 last 19.92) - open/close 15693 (time 1763338173.25 total 765.02 last 20.22) - open/close 15956 (time 1763338183.26 total 775.03 last 26.28) - open/close 16183 (time 1763338193.84 total 785.61 last 21.46) - open/close 16375 (time 1763338203.84 total 795.61 last 19.19) - open/close 16599 (time 1763338213.86 total 805.63 last 22.36) - open/close 16840 (time 1763338223.89 total 815.66 last 24.04) - open/close 17075 (time 1763338234.94 total 826.70 last 21.27) - open/close 17379 (time 1763338244.94 total 836.71 last 30.38) - open/close 18049 (time 1763338254.95 total 846.71 last 66.96) - open/close 18740 (time 1763338264.97 total 856.73 last 68.97) - open/close 19390 (time 1763338274.97 total 866.74 last 64.96) - open/close 20000 (time 1763338284.32 total 876.09 last 65.26) - open/close 20665 (time 1763338294.33 total 886.09 last 66.45) - open/close 21319 (time 1763338304.33 total 896.10 last 65.36) - open/close 22081 (time 1763338314.34 total 906.11 last 76.14) - open/close 22754 (time 1763338324.34 total 916.11 last 67.28) - open/close 23425 (time 1763338334.34 total 926.11 last 67.08) - open/close 24102 (time 1763338344.35 total 936.12 last 67.65) - open/close 24782 (time 1763338354.35 total 946.12 last 67.98) - open/close 25470 (time 1763338364.36 total 956.13 last 68.73) - open/close 26193 (time 1763338374.37 total 966.14 last 72.24) - open/close 26873 (time 1763338384.38 total 976.14 last 67.97) - open/close 27599 (time 1763338394.38 total 986.15 last 72.54) - open/close 28314 (time 1763338404.39 total 996.16 last 71.45) - open/close 29032 (time 1763338414.39 total 1006.16 last 71.78) - open/close 29676 (time 1763338424.40 total 1016.16 last 64.39) - open/close 30000 (time 1763338430.68 total 1022.45 last 51.55) - open/close 30668 (time 1763338440.70 total 1032.46 last 66.70) - open/close 31332 (time 1763338450.72 total 1042.49 last 66.20) - open/close 31999 (time 1763338460.73 total 1052.49 last 66.70) - open/close 32659 (time 1763338470.73 total 1062.50 last 65.94) - open/close 33261 (time 1763338480.74 total 1072.51 last 60.15) - open/close 33927 (time 1763338490.75 total 1082.52 last 66.54) - open/close 34567 (time 1763338500.77 total 1092.54 last 63.88) - open/close 35303 (time 1763338510.78 total 1102.55 last 73.49) - open/close 35917 (time 1763338520.80 total 1112.57 last 61.29) - open/close 36577 (time 1763338530.81 total 1122.58 last 65.94) - open/close 37232 (time 1763338540.81 total 1132.58 last 65.48) - open/close 37892 (time 1763338550.82 total 1142.59 last 65.94) - open/close 38559 (time 1763338560.83 total 1152.60 last 66.63) - open/close 39263 (time 1763338570.83 total 1162.60 last 70.39) - open/close 39995 (time 1763338580.86 total 1172.62 last 73.05) - open/close 40640 (time 1763338590.86 total 1182.63 last 64.45) - open/close 41276 (time 1763338600.88 total 1192.65 last 63.49) - open/close 41896 (time 1763338610.88 total 1202.65 last 62.00) - open/close 42560 (time 1763338620.89 total 1212.66 last 66.35) - open/close 43238 (time 1763338630.90 total 1222.67 last 67.71) - open/close 43914 (time 1763338640.90 total 1232.67 last 67.59) - open/close 44589 (time 1763338650.92 total 1242.68 last 67.41) - open/close 45305 (time 1763338660.92 total 1252.69 last 71.57) - open/close 46004 (time 1763338670.92 total 1262.69 last 69.87) - open/close 46693 (time 1763338680.93 total 1272.69 last 68.89) - open/close 47337 (time 1763338690.94 total 1282.71 last 64.30) - open/close 48009 (time 1763338700.95 total 1292.72 last 67.13) - open/close 48681 (time 1763338710.96 total 1302.73 last 67.13) - open/close 49315 (time 1763338720.98 total 1312.75 last 63.30) total: 50000 open/close in 1322.31 seconds: 37.81 ops/second - unlinked 0 (time 1763338734 ; total 0 ; last 0) - unlinked 10000 (time 1763338879 ; total 145 ; last 145) - unlinked 20000 (time 1763339020 ; total 286 ; last 141) - unlinked 30000 (time 1763339159 ; total 425 ; last 139) - unlinked 40000 (time 1763339298 ; total 564 ; last 139) total: 50000 unlinks in 701 seconds: 71.326675 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg320-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2