== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:59:22 (1764107962) start mds service on oleg134-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1764108006.40 total 10.01 last 18.29) - open/close 484 (time 1764108017.18 total 20.79 last 27.91) - open/close 818 (time 1764108028.06 total 31.67 last 30.70) - open/close 1126 (time 1764108038.50 total 42.11 last 29.51) - open/close 1433 (time 1764108048.88 total 52.49 last 29.58) - open/close 1633 (time 1764108058.91 total 62.51 last 19.95) - open/close 1748 (time 1764108073.33 total 76.94 last 7.97) - open/close 1978 (time 1764108083.34 total 86.94 last 23.00) - open/close 2183 (time 1764108093.93 total 97.54 last 19.35) - open/close 2265 (time 1764108104.29 total 107.90 last 7.92) - open/close 2453 (time 1764108115.18 total 118.78 last 17.27) - open/close 2627 (time 1764108125.28 total 128.89 last 17.21) - open/close 2833 (time 1764108135.66 total 139.26 last 19.86) - open/close 2975 (time 1764108145.90 total 149.51 last 13.86) - open/close 3137 (time 1764108156.46 total 160.07 last 15.34) - open/close 3244 (time 1764108167.27 total 170.87 last 9.90) - open/close 3412 (time 1764108177.83 total 181.44 last 15.90) - open/close 3621 (time 1764108188.61 total 192.22 last 19.39) - open/close 3865 (time 1764108199.28 total 202.88 last 22.88) - open/close 4105 (time 1764108209.32 total 212.93 last 23.89) - open/close 4301 (time 1764108219.88 total 223.49 last 18.56) - open/close 4483 (time 1764108229.90 total 233.50 last 18.18) - open/close 4712 (time 1764108239.91 total 243.51 last 22.87) - open/close 4951 (time 1764108249.91 total 253.52 last 23.88) - open/close 5193 (time 1764108260.22 total 263.83 last 23.48) - open/close 5409 (time 1764108271.08 total 274.69 last 19.89) - open/close 5625 (time 1764108281.84 total 285.45 last 20.07) - open/close 5839 (time 1764108292.46 total 296.06 last 20.16) - open/close 6015 (time 1764108302.46 total 306.07 last 17.59) - open/close 6199 (time 1764108313.20 total 316.81 last 17.13) - open/close 6332 (time 1764108323.24 total 326.85 last 13.25) - open/close 6485 (time 1764108333.61 total 337.22 last 14.75) - open/close 6672 (time 1764108343.66 total 347.27 last 18.61) - open/close 6814 (time 1764108353.84 total 357.44 last 13.95) - open/close 6963 (time 1764108363.88 total 367.48 last 14.84) - open/close 7163 (time 1764108373.88 total 377.49 last 19.99) - open/close 7328 (time 1764108384.55 total 388.16 last 15.47) - open/close 7495 (time 1764108395.38 total 398.99 last 15.42) - open/close 7611 (time 1764108405.48 total 409.08 last 11.49) - open/close 7786 (time 1764108415.67 total 419.28 last 17.17) - open/close 7945 (time 1764108426.27 total 429.88 last 14.99) - open/close 8115 (time 1764108437.24 total 440.85 last 15.50) - open/close 8252 (time 1764108447.53 total 451.14 last 13.31) - open/close 8443 (time 1764108458.47 total 462.08 last 17.46) - open/close 8590 (time 1764108468.64 total 472.25 last 14.45) - open/close 8729 (time 1764108478.95 total 482.56 last 13.49) - open/close 8878 (time 1764108489.13 total 492.74 last 14.64) - open/close 9072 (time 1764108499.14 total 502.74 last 19.39) - open/close 9189 (time 1764108509.94 total 513.55 last 10.82) - open/close 9329 (time 1764108520.88 total 524.48 last 12.81) - open/close 9443 (time 1764108531.50 total 535.10 last 10.74) - open/close 9599 (time 1764108541.54 total 545.15 last 15.53) - open/close 9748 (time 1764108551.79 total 555.39 last 14.55) - open/close 9941 (time 1764108562.05 total 565.66 last 18.80) - open/close 10139 (time 1764108572.05 total 575.66 last 19.79) - open/close 10358 (time 1764108582.97 total 586.57 last 20.07) - open/close 10595 (time 1764108592.98 total 596.58 last 23.67) - open/close 10828 (time 1764108602.98 total 606.59 last 23.29) - open/close 10976 (time 1764108613.28 total 616.89 last 14.37) - open/close 11147 (time 1764108624.17 total 627.77 last 15.71) - open/close 11302 (time 1764108634.73 total 638.34 last 14.67) - open/close 11416 (time 1764108645.36 total 648.96 last 10.73) - open/close 11536 (time 1764108656.05 total 659.66 last 11.22) - open/close 12475 (time 1764108666.06 total 669.67 last 93.78) - open/close 13410 (time 1764108676.08 total 679.68 last 93.37) - open/close 14261 (time 1764108686.09 total 689.69 last 85.02) - open/close 15218 (time 1764108696.10 total 699.71 last 95.58) - open/close 16168 (time 1764108706.11 total 709.72 last 94.89) - open/close 17172 (time 1764108716.13 total 719.73 last 100.25) - open/close 18145 (time 1764108726.13 total 729.74 last 97.24) - open/close 19125 (time 1764108736.14 total 739.75 last 97.94) - open/close 20000 (time 1764108744.56 total 748.17 last 103.87) - open/close 20946 (time 1764108754.58 total 758.19 last 94.46) - open/close 22030 (time 1764108764.58 total 768.19 last 108.40) - open/close 23187 (time 1764108774.58 total 778.19 last 115.65) - open/close 24291 (time 1764108784.59 total 788.20 last 110.33) - open/close 25244 (time 1764108794.59 total 798.20 last 95.30) - open/close 26218 (time 1764108804.60 total 808.20 last 97.33) - open/close 27295 (time 1764108814.60 total 818.21 last 107.69) - open/close 28449 (time 1764108824.60 total 828.21 last 115.37) - open/close 30000 (time 1764108834.15 total 837.75 last 162.47) - open/close 31627 (time 1764108844.15 total 847.76 last 162.65) - open/close 32767 (time 1764108854.15 total 857.76 last 113.98) - open/close 33776 (time 1764108864.16 total 867.76 last 100.86) - open/close 34623 (time 1764108874.16 total 877.77 last 84.66) - open/close 35480 (time 1764108884.17 total 887.78 last 85.64) - open/close 36555 (time 1764108894.17 total 897.78 last 107.48) - open/close 37454 (time 1764108904.18 total 907.79 last 89.81) - open/close 38292 (time 1764108914.18 total 917.79 last 83.79) - open/close 39243 (time 1764108924.19 total 927.79 last 95.05) - open/close 40000 (time 1764108931.18 total 934.78 last 108.30) - open/close 41084 (time 1764108941.18 total 944.79 last 108.32) - open/close 42046 (time 1764108951.19 total 954.80 last 96.09) - open/close 43160 (time 1764108961.20 total 964.81 last 111.34) - open/close 44306 (time 1764108971.21 total 974.81 last 114.51) - open/close 45471 (time 1764108981.21 total 984.82 last 116.49) - open/close 46804 (time 1764108991.21 total 994.82 last 133.28) - open/close 48423 (time 1764109001.21 total 1004.82 last 161.83) - open/close 49681 (time 1764109011.22 total 1014.83 last 125.72) total: 50000 open/close in 1017.31 seconds: 49.15 ops/second - unlinked 0 (time 1764109016 ; total 0 ; last 0) - unlinked 10000 (time 1764109127 ; total 111 ; last 111) - unlinked 20000 (time 1764109196 ; total 180 ; last 69) - unlinked 30000 (time 1764109255 ; total 239 ; last 59) - unlinked 40000 (time 1764109310 ; total 294 ; last 55) total: 50000 unlinks in 347 seconds: 144.092224 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg134-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2