== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:57:01 (1763348221) start mds service on oleg140-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1763348298.49 total 10.88 last 13.96) - open/close 372 (time 1763348309.37 total 21.77 last 20.21) - open/close 597 (time 1763348319.39 total 31.79 last 22.47) - open/close 813 (time 1763348330.18 total 42.57 last 20.02) - open/close 1073 (time 1763348340.18 total 52.58 last 25.99) - open/close 1199 (time 1763348350.20 total 62.60 last 12.57) - open/close 1372 (time 1763348360.22 total 72.62 last 17.26) - open/close 1555 (time 1763348370.26 total 82.66 last 18.22) - open/close 1768 (time 1763348380.82 total 93.22 last 20.18) - open/close 1917 (time 1763348390.82 total 103.22 last 14.90) - open/close 2027 (time 1763348400.89 total 113.29 last 10.92) - open/close 2273 (time 1763348411.65 total 124.05 last 22.87) - open/close 2515 (time 1763348422.38 total 134.78 last 22.55) - open/close 2758 (time 1763348433.40 total 145.80 last 22.05) - open/close 2998 (time 1763348444.28 total 156.68 last 22.06) - open/close 3213 (time 1763348454.30 total 166.69 last 21.47) - open/close 3420 (time 1763348464.31 total 176.71 last 20.67) - open/close 3591 (time 1763348474.80 total 187.20 last 16.29) - open/close 3834 (time 1763348484.83 total 197.23 last 24.24) - open/close 4097 (time 1763348495.15 total 207.55 last 25.47) - open/close 4371 (time 1763348505.20 total 217.60 last 27.27) - open/close 4520 (time 1763348515.23 total 227.62 last 14.86) - open/close 4768 (time 1763348525.23 total 237.63 last 24.78) - open/close 4916 (time 1763348535.69 total 248.09 last 14.16) - open/close 5132 (time 1763348546.05 total 258.45 last 20.85) - open/close 5344 (time 1763348556.74 total 269.14 last 19.82) - open/close 5487 (time 1763348566.96 total 279.36 last 13.99) - open/close 5761 (time 1763348576.97 total 289.36 last 27.40) - open/close 5991 (time 1763348587.58 total 299.98 last 21.67) - open/close 6196 (time 1763348598.21 total 310.61 last 19.29) - open/close 6417 (time 1763348608.21 total 320.61 last 22.08) - open/close 6591 (time 1763348618.23 total 330.63 last 17.37) - open/close 6780 (time 1763348628.73 total 341.13 last 18.01) - open/close 6987 (time 1763348638.73 total 351.13 last 20.69) - open/close 7218 (time 1763348649.12 total 361.52 last 22.24) - open/close 7443 (time 1763348660.10 total 372.50 last 20.48) - open/close 7707 (time 1763348670.83 total 383.23 last 24.60) - open/close 7890 (time 1763348680.84 total 393.23 last 18.30) - open/close 8122 (time 1763348690.84 total 403.24 last 23.18) - open/close 8352 (time 1763348700.84 total 413.24 last 23.00) - open/close 8562 (time 1763348710.85 total 423.25 last 20.99) - open/close 8762 (time 1763348721.14 total 433.54 last 19.44) - open/close 8949 (time 1763348731.39 total 443.79 last 18.25) - open/close 9153 (time 1763348741.87 total 454.27 last 19.46) - open/close 9336 (time 1763348751.98 total 464.38 last 18.09) - open/close 9565 (time 1763348761.99 total 474.39 last 22.87) - open/close 9874 (time 1763348772.01 total 484.41 last 30.85) - open/close 10000 (time 1763348777.42 total 489.82 last 23.29) - open/close 10147 (time 1763348788.22 total 500.61 last 13.62) - open/close 10350 (time 1763348799.28 total 511.67 last 18.35) - open/close 10536 (time 1763348809.52 total 521.92 last 18.15) - open/close 10679 (time 1763348819.57 total 531.97 last 14.23) - open/close 10908 (time 1763348830.00 total 542.40 last 21.96) - open/close 11134 (time 1763348840.02 total 552.42 last 22.56) - open/close 11364 (time 1763348851.06 total 563.46 last 20.83) - open/close 11594 (time 1763348861.08 total 573.47 last 22.96) - open/close 11819 (time 1763348871.08 total 583.48 last 22.49) - open/close 12026 (time 1763348881.80 total 594.20 last 19.32) - open/close 12278 (time 1763348892.81 total 605.21 last 22.88) - open/close 12471 (time 1763348903.48 total 615.88 last 18.09) - open/close 12736 (time 1763348913.50 total 625.89 last 26.46) - open/close 12936 (time 1763348923.57 total 635.97 last 19.86) - open/close 13175 (time 1763348933.57 total 645.97 last 23.89) - open/close 13386 (time 1763348943.58 total 655.98 last 21.08) - open/close 13589 (time 1763348955.65 total 668.05 last 16.82) - open/close 13781 (time 1763348966.14 total 678.54 last 18.31) - open/close 13946 (time 1763348976.19 total 688.59 last 16.41) - open/close 14163 (time 1763348986.20 total 698.60 last 21.68) - open/close 14337 (time 1763348999.04 total 711.44 last 13.55) - open/close 14556 (time 1763349009.05 total 721.45 last 21.88) - open/close 14730 (time 1763349019.82 total 732.22 last 16.15) - open/close 14974 (time 1763349029.84 total 742.24 last 24.35) - open/close 15095 (time 1763349040.34 total 752.74 last 11.53) - open/close 15291 (time 1763349051.27 total 763.66 last 17.94) - open/close 15510 (time 1763349061.27 total 773.67 last 21.90) - open/close 15746 (time 1763349071.28 total 783.68 last 23.58) - open/close 15906 (time 1763349081.29 total 793.69 last 15.98) - open/close 16051 (time 1763349091.77 total 804.16 last 13.84) - open/close 16260 (time 1763349102.26 total 814.65 last 19.92) - open/close 16494 (time 1763349113.28 total 825.67 last 21.23) - open/close 16920 (time 1763349123.28 total 835.68 last 42.57) - open/close 17670 (time 1763349133.29 total 845.69 last 74.95) - open/close 18382 (time 1763349143.30 total 855.70 last 71.13) - open/close 19065 (time 1763349153.30 total 865.70 last 68.30) - open/close 19728 (time 1763349163.30 total 875.70 last 66.28) - open/close 20000 (time 1763349166.94 total 879.34 last 74.74) - open/close 20690 (time 1763349176.94 total 889.34 last 68.98) - open/close 21401 (time 1763349186.95 total 899.35 last 71.08) - open/close 22073 (time 1763349196.98 total 909.38 last 67.00) - open/close 22800 (time 1763349206.98 total 919.38 last 72.67) - open/close 23449 (time 1763349216.98 total 929.38 last 64.88) - open/close 24102 (time 1763349226.99 total 939.39 last 65.28) - open/close 24736 (time 1763349237.00 total 949.40 last 63.32) - open/close 25366 (time 1763349247.01 total 959.41 last 62.93) - open/close 26021 (time 1763349257.02 total 969.42 last 65.42) - open/close 26664 (time 1763349267.03 total 979.43 last 64.23) - open/close 27334 (time 1763349277.04 total 989.44 last 66.95) - open/close 28018 (time 1763349287.04 total 999.44 last 68.38) - open/close 28721 (time 1763349297.06 total 1009.46 last 70.18) - open/close 29384 (time 1763349307.06 total 1019.46 last 66.29) - open/close 30000 (time 1763349316.54 total 1028.93 last 65.02) - open/close 30669 (time 1763349326.54 total 1038.94 last 66.89) - open/close 31390 (time 1763349336.55 total 1048.95 last 71.99) - open/close 32067 (time 1763349346.56 total 1058.96 last 67.62) - open/close 32739 (time 1763349356.57 total 1068.97 last 67.14) - open/close 33397 (time 1763349366.58 total 1078.98 last 65.73) - open/close 34084 (time 1763349376.59 total 1088.99 last 68.67) - open/close 34742 (time 1763349386.61 total 1099.01 last 65.67) - open/close 35434 (time 1763349396.61 total 1109.01 last 69.16) - open/close 36224 (time 1763349406.62 total 1119.02 last 78.93) - open/close 36964 (time 1763349416.63 total 1129.03 last 73.96) - open/close 37631 (time 1763349426.64 total 1139.04 last 66.63) - open/close 38164 (time 1763349436.64 total 1149.04 last 53.27) - open/close 38833 (time 1763349446.65 total 1159.05 last 66.87) - open/close 39419 (time 1763349456.66 total 1169.06 last 58.53) - open/close 40000 (time 1763349465.39 total 1177.79 last 66.54) - open/close 40714 (time 1763349475.40 total 1187.80 last 71.35) - open/close 41421 (time 1763349485.40 total 1197.80 last 70.65) - open/close 42121 (time 1763349495.41 total 1207.81 last 69.93) - open/close 42763 (time 1763349505.42 total 1217.82 last 64.19) - open/close 43376 (time 1763349515.42 total 1227.82 last 61.25) - open/close 44039 (time 1763349525.44 total 1237.84 last 66.19) - open/close 44629 (time 1763349535.45 total 1247.85 last 58.96) - open/close 45316 (time 1763349545.45 total 1257.85 last 68.69) - open/close 45942 (time 1763349555.47 total 1267.87 last 62.46) - open/close 46613 (time 1763349565.48 total 1277.88 last 67.08) - open/close 47273 (time 1763349575.49 total 1287.89 last 65.93) - open/close 47940 (time 1763349585.56 total 1297.96 last 66.24) - open/close 48594 (time 1763349595.57 total 1307.97 last 65.29) - open/close 49269 (time 1763349605.58 total 1317.98 last 67.43) - open/close 49942 (time 1763349615.58 total 1327.98 last 67.29) total: 50000 open/close in 1328.80 seconds: 37.63 ops/second - unlinked 0 (time 1763349619 ; total 0 ; last 0) - unlinked 10000 (time 1763349762 ; total 143 ; last 143) - unlinked 20000 (time 1763349909 ; total 290 ; last 147) - unlinked 30000 (time 1763350052 ; total 433 ; last 143) - unlinked 40000 (time 1763350195 ; total 576 ; last 143) total: 50000 unlinks in 718 seconds: 69.637886 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2