== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:31:19 (1763346679) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 227 (time 1763346752.60 total 10.56 last 21.50) - open/close 450 (time 1763346763.61 total 21.58 last 20.24) - open/close 690 (time 1763346774.17 total 32.13 last 22.74) - open/close 933 (time 1763346784.84 total 42.81 last 22.76) - open/close 1146 (time 1763346794.84 total 52.81 last 21.30) - open/close 1350 (time 1763346804.86 total 62.82 last 20.38) - open/close 1549 (time 1763346814.86 total 72.82 last 19.90) - open/close 1742 (time 1763346824.86 total 82.82 last 19.30) - open/close 2016 (time 1763346834.90 total 92.86 last 27.30) - open/close 2291 (time 1763346844.91 total 102.87 last 27.47) - open/close 2482 (time 1763346855.45 total 113.41 last 18.12) - open/close 2681 (time 1763346865.45 total 123.42 last 19.89) - open/close 2884 (time 1763346876.19 total 134.15 last 18.90) - open/close 3100 (time 1763346886.44 total 144.40 last 21.08) - open/close 3342 (time 1763346896.45 total 154.41 last 24.18) - open/close 3517 (time 1763346906.53 total 164.49 last 17.35) - open/close 3697 (time 1763346916.54 total 174.50 last 17.99) - open/close 3923 (time 1763346927.20 total 185.16 last 21.20) - open/close 4174 (time 1763346937.20 total 195.17 last 25.08) - open/close 4382 (time 1763346947.21 total 205.17 last 20.80) - open/close 4580 (time 1763346957.22 total 215.19 last 19.76) - open/close 4792 (time 1763346967.23 total 225.19 last 21.19) - open/close 5003 (time 1763346977.24 total 235.20 last 21.08) - open/close 5282 (time 1763346987.93 total 245.90 last 26.09) - open/close 5486 (time 1763346998.49 total 256.45 last 19.33) - open/close 5743 (time 1763347009.07 total 267.03 last 24.28) - open/close 5973 (time 1763347019.07 total 277.04 last 23.00) - open/close 6184 (time 1763347029.33 total 287.29 last 20.58) - open/close 6385 (time 1763347039.96 total 297.92 last 18.91) - open/close 6582 (time 1763347050.26 total 308.22 last 19.13) - open/close 6800 (time 1763347060.45 total 318.42 last 21.38) - open/close 6978 (time 1763347071.20 total 329.16 last 16.57) - open/close 7197 (time 1763347081.20 total 339.17 last 21.89) - open/close 7428 (time 1763347091.21 total 349.17 last 23.09) - open/close 7650 (time 1763347101.21 total 359.18 last 22.18) - open/close 7841 (time 1763347112.08 total 370.04 last 17.58) - open/close 8068 (time 1763347122.10 total 380.07 last 22.65) - open/close 8172 (time 1763347132.94 total 390.91 last 9.59) - open/close 8413 (time 1763347143.53 total 401.49 last 22.77) - open/close 8620 (time 1763347153.54 total 411.51 last 20.67) - open/close 8823 (time 1763347164.44 total 422.40 last 18.63) - open/close 9070 (time 1763347175.06 total 433.02 last 23.26) - open/close 9286 (time 1763347185.07 total 443.03 last 21.57) - open/close 9466 (time 1763347195.67 total 453.64 last 16.97) - open/close 9690 (time 1763347206.69 total 464.65 last 20.34) - open/close 9899 (time 1763347217.30 total 475.26 last 19.70) - open/close 10000 (time 1763347223.12 total 481.09 last 17.33) - open/close 10157 (time 1763347233.95 total 491.91 last 14.51) - open/close 10368 (time 1763347244.90 total 502.86 last 19.27) - open/close 10565 (time 1763347255.26 total 513.22 last 19.01) - open/close 10762 (time 1763347265.27 total 523.23 last 19.68) - open/close 10977 (time 1763347276.19 total 534.15 last 19.69) - open/close 11174 (time 1763347286.82 total 544.78 last 18.54) - open/close 11341 (time 1763347297.61 total 555.58 last 15.47) - open/close 11481 (time 1763347307.99 total 565.95 last 13.50) - open/close 11643 (time 1763347318.00 total 575.96 last 16.18) - open/close 11817 (time 1763347328.30 total 586.26 last 16.89) - open/close 12021 (time 1763347339.34 total 597.31 last 18.47) - open/close 12249 (time 1763347349.34 total 607.31 last 22.80) - open/close 12416 (time 1763347359.54 total 617.50 last 16.38) - open/close 12632 (time 1763347370.51 total 628.47 last 19.69) - open/close 12812 (time 1763347380.52 total 638.48 last 17.97) - open/close 12979 (time 1763347390.55 total 648.51 last 16.66) - open/close 13156 (time 1763347400.66 total 658.62 last 17.51) - open/close 13279 (time 1763347411.67 total 669.63 last 11.17) - open/close 13450 (time 1763347422.55 total 680.51 last 15.72) - open/close 13564 (time 1763347433.44 total 691.41 last 10.46) - open/close 13678 (time 1763347444.50 total 702.46 last 10.31) - open/close 13814 (time 1763347454.93 total 712.89 last 13.04) - open/close 13954 (time 1763347465.05 total 723.01 last 13.84) - open/close 14152 (time 1763347475.36 total 733.32 last 19.20) - open/close 14353 (time 1763347485.37 total 743.33 last 20.08) - open/close 14533 (time 1763347495.57 total 753.53 last 17.64) - open/close 14642 (time 1763347506.32 total 764.28 last 10.14) - open/close 14760 (time 1763347516.32 total 774.29 last 11.79) - open/close 15429 (time 1763347526.33 total 784.29 last 66.86) - open/close 16144 (time 1763347536.33 total 794.30 last 71.47) - open/close 16851 (time 1763347546.34 total 804.31 last 70.64) - open/close 17494 (time 1763347556.35 total 814.31 last 64.27) - open/close 18192 (time 1763347566.37 total 824.33 last 69.65) - open/close 18939 (time 1763347576.37 total 834.34 last 74.65) - open/close 19709 (time 1763347586.38 total 844.35 last 76.94) - open/close 20000 (time 1763347590.32 total 848.28 last 73.97) - open/close 20866 (time 1763347600.32 total 858.28 last 86.58) - open/close 21684 (time 1763347610.32 total 868.29 last 81.77) - open/close 22410 (time 1763347620.33 total 878.30 last 72.52) - open/close 23148 (time 1763347630.34 total 888.30 last 73.75) - open/close 23908 (time 1763347640.35 total 898.31 last 75.92) - open/close 24635 (time 1763347650.37 total 908.34 last 72.54) - open/close 25459 (time 1763347660.38 total 918.34 last 82.34) - open/close 26290 (time 1763347670.39 total 928.35 last 83.04) - open/close 27073 (time 1763347680.39 total 938.36 last 78.26) - open/close 27761 (time 1763347690.41 total 948.37 last 68.71) - open/close 28501 (time 1763347700.41 total 958.37 last 73.97) - open/close 29291 (time 1763347710.42 total 968.38 last 78.95) - open/close 30000 (time 1763347720.02 total 977.99 last 73.80) - open/close 30789 (time 1763347730.03 total 988.00 last 78.83) - open/close 31556 (time 1763347740.03 total 998.00 last 76.69) - open/close 32428 (time 1763347750.04 total 1008.01 last 87.12) - open/close 33283 (time 1763347760.04 total 1018.01 last 85.49) - open/close 34158 (time 1763347770.04 total 1028.01 last 87.49) - open/close 34982 (time 1763347780.05 total 1038.02 last 82.31) - open/close 35741 (time 1763347790.06 total 1048.02 last 75.86) - open/close 36570 (time 1763347800.07 total 1058.03 last 82.85) - open/close 37288 (time 1763347810.07 total 1068.03 last 71.77) - open/close 38075 (time 1763347820.09 total 1078.06 last 78.51) - open/close 38886 (time 1763347830.10 total 1088.07 last 81.04) - open/close 39604 (time 1763347840.10 total 1098.07 last 71.78) - open/close 40000 (time 1763347845.04 total 1103.00 last 80.23) - open/close 40787 (time 1763347855.05 total 1113.01 last 78.62) - open/close 41516 (time 1763347865.06 total 1123.02 last 72.85) - open/close 42191 (time 1763347875.07 total 1133.04 last 67.38) - open/close 42920 (time 1763347885.08 total 1143.05 last 72.84) - open/close 43635 (time 1763347895.09 total 1153.05 last 71.45) - open/close 44407 (time 1763347905.10 total 1163.06 last 77.16) - open/close 45120 (time 1763347915.10 total 1173.06 last 71.27) - open/close 45859 (time 1763347925.12 total 1183.09 last 73.72) - open/close 46570 (time 1763347935.13 total 1193.09 last 71.09) - open/close 47243 (time 1763347945.14 total 1203.11 last 67.17) - open/close 47904 (time 1763347955.15 total 1213.11 last 66.08) - open/close 48630 (time 1763347965.15 total 1223.11 last 72.59) - open/close 49379 (time 1763347975.15 total 1233.12 last 74.87) total: 50000 open/close in 1241.47 seconds: 40.27 ops/second - unlinked 0 (time 1763347986 ; total 0 ; last 0) - unlinked 10000 (time 1763348120 ; total 134 ; last 134) - unlinked 20000 (time 1763348235 ; total 249 ; last 115) - unlinked 30000 (time 1763348361 ; total 375 ; last 126) - unlinked 40000 (time 1763348484 ; total 498 ; last 123) total: 50000 unlinks in 619 seconds: 80.775444 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2