== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:45:15 (1768254315) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1768254405.41 total 10.90 last 22.02) - open/close 472 (time 1768254416.35 total 21.84 last 21.20) - open/close 742 (time 1768254427.16 total 32.65 last 24.99) - open/close 981 (time 1768254438.18 total 43.67 last 21.67) - open/close 1237 (time 1768254448.19 total 53.68 last 25.57) - open/close 1474 (time 1768254461.21 total 66.70 last 18.21) - open/close 1653 (time 1768254471.65 total 77.14 last 17.15) - open/close 1864 (time 1768254482.42 total 87.91 last 19.59) - open/close 2024 (time 1768254492.46 total 97.95 last 15.94) - open/close 2281 (time 1768254502.46 total 107.95 last 25.69) - open/close 2514 (time 1768254513.18 total 118.67 last 21.74) - open/close 2770 (time 1768254523.19 total 128.67 last 25.59) - open/close 3017 (time 1768254533.19 total 138.68 last 24.68) - open/close 3290 (time 1768254543.21 total 148.70 last 27.24) - open/close 3576 (time 1768254553.59 total 159.08 last 27.57) - open/close 3832 (time 1768254564.63 total 170.12 last 23.18) - open/close 4112 (time 1768254574.65 total 180.14 last 27.95) - open/close 4353 (time 1768254584.67 total 190.16 last 24.05) - open/close 4605 (time 1768254594.69 total 200.18 last 25.15) - open/close 4839 (time 1768254604.70 total 210.19 last 23.38) - open/close 5098 (time 1768254615.06 total 220.55 last 25.01) - open/close 5363 (time 1768254625.31 total 230.80 last 25.84) - open/close 5622 (time 1768254635.39 total 240.88 last 25.70) - open/close 5861 (time 1768254645.39 total 250.88 last 23.90) - open/close 6114 (time 1768254656.43 total 261.92 last 22.92) - open/close 6328 (time 1768254666.45 total 271.94 last 21.36) - open/close 6501 (time 1768254676.58 total 282.06 last 17.09) - open/close 6702 (time 1768254687.57 total 293.06 last 18.28) - open/close 6838 (time 1768254697.57 total 303.06 last 13.60) - open/close 6989 (time 1768254707.87 total 313.36 last 14.67) - open/close 7132 (time 1768254718.24 total 323.73 last 13.79) - open/close 7374 (time 1768254728.54 total 334.03 last 23.50) - open/close 7590 (time 1768254739.49 total 344.98 last 19.71) - open/close 7783 (time 1768254749.72 total 355.21 last 18.87) - open/close 8035 (time 1768254760.61 total 366.10 last 23.14) - open/close 8275 (time 1768254770.63 total 376.12 last 23.96) - open/close 8433 (time 1768254780.64 total 386.13 last 15.78) - open/close 8566 (time 1768254790.69 total 396.18 last 13.23) - open/close 8731 (time 1768254800.98 total 406.47 last 16.03) - open/close 8864 (time 1768254810.99 total 416.48 last 13.29) - open/close 9002 (time 1768254821.01 total 426.50 last 13.77) - open/close 9157 (time 1768254831.20 total 436.69 last 15.22) - open/close 9325 (time 1768254841.21 total 446.70 last 16.77) - open/close 9509 (time 1768254852.22 total 457.71 last 16.71) - open/close 9712 (time 1768254862.39 total 467.88 last 19.95) - open/close 9836 (time 1768254873.12 total 478.61 last 11.57) - open/close 10000 (time 1768254881.04 total 486.53 last 20.69) - open/close 10194 (time 1768254891.05 total 496.54 last 19.39) - open/close 10268 (time 1768254901.74 total 507.23 last 6.92) - open/close 10337 (time 1768254912.03 total 517.52 last 6.71) - open/close 10506 (time 1768254922.14 total 527.63 last 16.71) - open/close 10689 (time 1768254932.14 total 537.63 last 18.30) - open/close 10923 (time 1768254944.73 total 550.22 last 18.59) - open/close 11155 (time 1768254954.74 total 560.22 last 23.20) - open/close 11404 (time 1768254965.63 total 571.12 last 22.84) - open/close 11641 (time 1768254975.80 total 581.29 last 23.32) - open/close 11837 (time 1768254986.34 total 591.83 last 18.59) - open/close 12003 (time 1768254996.38 total 601.87 last 16.54) - open/close 12309 (time 1768255006.38 total 611.87 last 30.58) - open/close 12424 (time 1768255016.68 total 622.17 last 11.17) - open/close 12473 (time 1768255026.69 total 632.18 last 4.89) - open/close 12579 (time 1768255036.71 total 642.20 last 10.58) - open/close 12793 (time 1768255046.72 total 652.21 last 21.40) - open/close 13007 (time 1768255056.75 total 662.24 last 21.32) - open/close 13253 (time 1768255067.56 total 673.05 last 22.77) - open/close 13476 (time 1768255080.22 total 685.71 last 17.61) - open/close 13707 (time 1768255090.92 total 696.41 last 21.60) - open/close 13971 (time 1768255101.99 total 707.48 last 23.85) - open/close 14199 (time 1768255112.42 total 717.91 last 21.85) - open/close 14407 (time 1768255122.44 total 727.93 last 20.77) - open/close 14630 (time 1768255133.28 total 738.77 last 20.58) - open/close 14937 (time 1768255143.30 total 748.79 last 30.63) - open/close 15165 (time 1768255154.16 total 759.65 last 21.00) - open/close 15337 (time 1768255164.16 total 769.65 last 17.20) - open/close 15556 (time 1768255174.70 total 780.19 last 20.78) - open/close 15765 (time 1768255184.71 total 790.20 last 20.88) - open/close 15957 (time 1768255195.20 total 800.69 last 18.30) - open/close 16153 (time 1768255205.52 total 811.01 last 18.99) - open/close 16345 (time 1768255216.30 total 821.79 last 17.81) - open/close 16537 (time 1768255226.32 total 831.81 last 19.18) - open/close 16773 (time 1768255236.32 total 841.81 last 23.59) - open/close 16972 (time 1768255246.33 total 851.81 last 19.89) - open/close 17235 (time 1768255256.33 total 861.82 last 26.30) - open/close 17811 (time 1768255266.33 total 871.82 last 57.60) - open/close 18646 (time 1768255276.33 total 881.82 last 83.47) - open/close 19450 (time 1768255286.34 total 891.83 last 80.34) - open/close 20000 (time 1768255293.13 total 898.62 last 80.95) - open/close 20876 (time 1768255303.20 total 908.69 last 87.03) - open/close 21656 (time 1768255313.20 total 918.69 last 77.96) - open/close 22331 (time 1768255323.22 total 928.71 last 67.38) - open/close 23196 (time 1768255333.22 total 938.71 last 86.47) - open/close 24103 (time 1768255343.22 total 948.71 last 90.68) - open/close 24866 (time 1768255353.23 total 958.72 last 76.25) - open/close 25606 (time 1768255363.23 total 968.72 last 73.99) - open/close 26273 (time 1768255373.23 total 978.72 last 66.69) - open/close 27187 (time 1768255383.25 total 988.74 last 91.27) - open/close 28081 (time 1768255393.25 total 998.74 last 89.39) - open/close 29082 (time 1768255403.25 total 1008.74 last 100.05) - open/close 29988 (time 1768255413.26 total 1018.75 last 90.55) - open/close 30938 (time 1768255423.26 total 1028.75 last 94.96) - open/close 31751 (time 1768255433.27 total 1038.76 last 81.27) - open/close 32689 (time 1768255443.27 total 1048.76 last 93.74) - open/close 33577 (time 1768255453.27 total 1058.76 last 88.79) - open/close 34304 (time 1768255463.28 total 1068.77 last 72.65) - open/close 35115 (time 1768255473.29 total 1078.78 last 81.02) - open/close 35889 (time 1768255483.31 total 1088.80 last 77.25) - open/close 36633 (time 1768255493.31 total 1098.80 last 74.38) - open/close 37495 (time 1768255503.32 total 1108.81 last 86.15) - open/close 38333 (time 1768255513.32 total 1118.81 last 83.79) - open/close 39129 (time 1768255523.35 total 1128.84 last 79.38) - open/close 39977 (time 1768255533.35 total 1138.84 last 84.80) - open/close 40789 (time 1768255543.36 total 1148.85 last 81.13) - open/close 41715 (time 1768255553.36 total 1158.85 last 92.59) - open/close 42532 (time 1768255563.36 total 1168.85 last 81.69) - open/close 43452 (time 1768255573.36 total 1178.85 last 91.95) - open/close 44230 (time 1768255583.36 total 1188.85 last 77.79) - open/close 45081 (time 1768255593.37 total 1198.86 last 85.08) - open/close 45937 (time 1768255603.37 total 1208.86 last 85.57) - open/close 46789 (time 1768255613.38 total 1218.87 last 85.13) - open/close 47683 (time 1768255623.38 total 1228.87 last 89.36) - open/close 48570 (time 1768255633.39 total 1238.88 last 88.64) - open/close 49329 (time 1768255643.40 total 1248.89 last 75.86) total: 50000 open/close in 1257.35 seconds: 39.77 ops/second - unlinked 0 (time 1768255655 ; total 0 ; last 0) - unlinked 10000 (time 1768255767 ; total 112 ; last 112) - unlinked 20000 (time 1768255862 ; total 207 ; last 95) - unlinked 30000 (time 1768255924 ; total 269 ; last 62) - unlinked 40000 (time 1768255984 ; total 329 ; last 60) total: 50000 unlinks in 389 seconds: 128.534698 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2