== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:41:01 (1768254061) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 267 (time 1768254132.15 total 10.83 last 24.65) - open/close 511 (time 1768254142.15 total 20.83 last 24.39) - open/close 755 (time 1768254152.99 total 31.67 last 22.52) - open/close 877 (time 1768254164.00 total 42.69 last 11.08) - open/close 1137 (time 1768254174.69 total 53.38 last 24.32) - open/close 1372 (time 1768254184.70 total 63.39 last 23.48) - open/close 1563 (time 1768254194.72 total 73.40 last 19.07) - open/close 1729 (time 1768254204.72 total 83.41 last 16.59) - open/close 1952 (time 1768254214.73 total 93.41 last 22.29) - open/close 2171 (time 1768254225.57 total 104.26 last 20.19) - open/close 2401 (time 1768254235.75 total 114.43 last 22.61) - open/close 2658 (time 1768254246.75 total 125.43 last 23.37) - open/close 2855 (time 1768254256.81 total 135.49 last 19.58) - open/close 3089 (time 1768254266.81 total 145.49 last 23.39) - open/close 3274 (time 1768254277.16 total 155.84 last 17.88) - open/close 3458 (time 1768254287.33 total 166.02 last 18.08) - open/close 3688 (time 1768254297.76 total 176.45 last 22.05) - open/close 3963 (time 1768254307.89 total 186.57 last 27.16) - open/close 4194 (time 1768254318.81 total 197.49 last 21.16) - open/close 4401 (time 1768254329.24 total 207.93 last 19.83) - open/close 4565 (time 1768254339.25 total 217.93 last 16.39) - open/close 4758 (time 1768254349.73 total 228.41 last 18.41) - open/close 4995 (time 1768254359.73 total 238.41 last 23.70) - open/close 5226 (time 1768254370.21 total 248.89 last 22.04) - open/close 5468 (time 1768254380.51 total 259.19 last 23.49) - open/close 5662 (time 1768254390.81 total 269.49 last 18.84) - open/close 5838 (time 1768254401.06 total 279.74 last 17.17) - open/close 6073 (time 1768254411.81 total 290.49 last 21.86) - open/close 6291 (time 1768254422.10 total 300.79 last 21.17) - open/close 6489 (time 1768254433.12 total 311.80 last 17.98) - open/close 6651 (time 1768254443.96 total 322.65 last 14.94) - open/close 6891 (time 1768254454.82 total 333.50 last 22.11) - open/close 7071 (time 1768254464.83 total 343.51 last 17.99) - open/close 7271 (time 1768254474.83 total 353.51 last 19.99) - open/close 7456 (time 1768254485.74 total 364.42 last 16.97) - open/close 7594 (time 1768254495.96 total 374.64 last 13.49) - open/close 7818 (time 1768254505.99 total 384.67 last 22.34) - open/close 8033 (time 1768254516.89 total 395.57 last 19.73) - open/close 8197 (time 1768254527.39 total 406.07 last 15.62) - open/close 8367 (time 1768254537.58 total 416.26 last 16.68) - open/close 8665 (time 1768254547.59 total 426.27 last 29.78) - open/close 8880 (time 1768254557.80 total 436.48 last 21.05) - open/close 9094 (time 1768254568.66 total 447.35 last 19.70) - open/close 9295 (time 1768254578.67 total 457.35 last 20.08) - open/close 9484 (time 1768254588.69 total 467.37 last 18.86) - open/close 9669 (time 1768254598.75 total 477.43 last 18.40) - open/close 9860 (time 1768254608.76 total 487.45 last 19.07) - open/close 10000 (time 1768254615.87 total 494.56 last 19.69) - open/close 10215 (time 1768254626.78 total 505.47 last 19.71) - open/close 10420 (time 1768254637.02 total 515.70 last 20.03) - open/close 10673 (time 1768254647.85 total 526.54 last 23.35) - open/close 10855 (time 1768254658.16 total 536.85 last 17.65) - open/close 11065 (time 1768254668.17 total 546.85 last 20.99) - open/close 11218 (time 1768254678.88 total 557.57 last 14.28) - open/close 11443 (time 1768254689.37 total 568.05 last 21.45) - open/close 11644 (time 1768254701.49 total 580.17 last 16.58) - open/close 11834 (time 1768254711.50 total 590.19 last 18.98) - open/close 12030 (time 1768254722.34 total 601.03 last 18.08) - open/close 12338 (time 1768254732.45 total 611.13 last 30.48) - open/close 12656 (time 1768254743.08 total 621.76 last 29.93) - open/close 12865 (time 1768254753.37 total 632.06 last 20.29) - open/close 13085 (time 1768254764.19 total 642.87 last 20.35) - open/close 13238 (time 1768254774.68 total 653.37 last 14.58) - open/close 13437 (time 1768254784.99 total 663.68 last 19.30) - open/close 13642 (time 1768254795.18 total 673.86 last 20.13) - open/close 13876 (time 1768254805.19 total 683.87 last 23.37) - open/close 14014 (time 1768254815.40 total 694.08 last 13.51) - open/close 14255 (time 1768254825.41 total 704.10 last 24.07) - open/close 14508 (time 1768254835.42 total 714.10 last 25.28) - open/close 15054 (time 1768254845.42 total 724.10 last 54.60) - open/close 15398 (time 1768254855.44 total 734.12 last 34.33) - open/close 15896 (time 1768254865.44 total 744.13 last 49.80) - open/close 16423 (time 1768254875.49 total 754.18 last 52.44) - open/close 16792 (time 1768254885.52 total 764.20 last 36.81) - open/close 17190 (time 1768254895.53 total 774.22 last 39.74) - open/close 17650 (time 1768254905.54 total 784.23 last 45.96) - open/close 18233 (time 1768254915.57 total 794.25 last 58.14) - open/close 18715 (time 1768254925.58 total 804.26 last 48.17) - open/close 19275 (time 1768254935.58 total 814.27 last 55.96) - open/close 19938 (time 1768254945.59 total 824.27 last 66.25) - open/close 20680 (time 1768254955.60 total 834.28 last 74.14) - open/close 21387 (time 1768254965.63 total 844.31 last 70.49) - open/close 22017 (time 1768254975.65 total 854.33 last 62.89) - open/close 22653 (time 1768254985.65 total 864.34 last 63.56) - open/close 23330 (time 1768254995.69 total 874.38 last 67.43) - open/close 24058 (time 1768255005.71 total 884.39 last 72.68) - open/close 24644 (time 1768255015.72 total 894.40 last 58.55) - open/close 25346 (time 1768255025.72 total 904.41 last 70.16) - open/close 26005 (time 1768255035.73 total 914.42 last 65.84) - open/close 26601 (time 1768255045.74 total 924.43 last 59.55) - open/close 27192 (time 1768255055.76 total 934.44 last 59.02) - open/close 27905 (time 1768255065.76 total 944.45 last 71.25) - open/close 28638 (time 1768255075.77 total 954.45 last 73.28) - open/close 29286 (time 1768255085.78 total 964.46 last 64.74) - open/close 29924 (time 1768255095.78 total 974.47 last 63.75) - open/close 30591 (time 1768255105.78 total 984.47 last 66.70) - open/close 31294 (time 1768255115.79 total 994.47 last 70.28) - open/close 31925 (time 1768255125.79 total 1004.47 last 63.08) - open/close 32552 (time 1768255135.82 total 1014.50 last 62.52) - open/close 33112 (time 1768255145.82 total 1024.51 last 55.96) - open/close 33832 (time 1768255155.83 total 1034.51 last 71.98) - open/close 34540 (time 1768255165.84 total 1044.53 last 70.69) - open/close 35196 (time 1768255175.86 total 1054.54 last 65.50) - open/close 35808 (time 1768255185.87 total 1064.55 last 61.15) - open/close 36501 (time 1768255195.89 total 1074.57 last 69.18) - open/close 37173 (time 1768255205.97 total 1084.65 last 66.66) - open/close 37828 (time 1768255215.98 total 1094.67 last 65.38) - open/close 38503 (time 1768255226.00 total 1104.69 last 67.38) - open/close 39183 (time 1768255236.01 total 1114.69 last 67.96) - open/close 39886 (time 1768255246.03 total 1124.72 last 70.12) - open/close 40000 (time 1768255248.25 total 1126.93 last 51.38) - open/close 40596 (time 1768255258.26 total 1136.94 last 59.55) - open/close 41271 (time 1768255268.27 total 1146.95 last 67.46) - open/close 41952 (time 1768255278.28 total 1156.96 last 68.02) - open/close 42667 (time 1768255288.29 total 1166.97 last 71.44) - open/close 43347 (time 1768255298.29 total 1176.97 last 67.97) - open/close 43959 (time 1768255308.33 total 1187.01 last 60.96) - open/close 44553 (time 1768255318.34 total 1197.03 last 59.32) - open/close 45145 (time 1768255328.35 total 1207.03 last 59.19) - open/close 45751 (time 1768255338.36 total 1217.05 last 60.50) - open/close 46463 (time 1768255348.37 total 1227.05 last 71.14) - open/close 47172 (time 1768255358.37 total 1237.06 last 70.89) - open/close 47893 (time 1768255368.38 total 1247.06 last 72.06) - open/close 48630 (time 1768255378.38 total 1257.06 last 73.70) - open/close 49351 (time 1768255388.39 total 1267.07 last 72.02) total: 50000 open/close in 1276.76 seconds: 39.16 ops/second - unlinked 0 (time 1768255402 ; total 0 ; last 0) - unlinked 10000 (time 1768255546 ; total 144 ; last 144) - unlinked 20000 (time 1768255674 ; total 272 ; last 128) - unlinked 30000 (time 1768255788 ; total 386 ; last 114) - unlinked 40000 (time 1768255899 ; total 497 ; last 111) total: 50000 unlinks in 602 seconds: 83.056480 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg406-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2