== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:33:31 (1787855611) start mds service on oleg326-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions oleg326-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg326-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg326-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg326-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1787855685.93 total 10.01 last 20.99) - open/close 374 (time 1787855696.38 total 20.46 last 15.70) - open/close 508 (time 1787855707.25 total 31.33 last 12.33) - open/close 701 (time 1787855717.36 total 41.43 last 19.10) - open/close 855 (time 1787855728.23 total 52.31 last 14.16) - open/close 983 (time 1787855738.26 total 62.33 last 12.77) - open/close 1128 (time 1787855749.05 total 73.12 last 13.44) - open/close 1345 (time 1787855759.29 total 83.36 last 21.20) - open/close 1522 (time 1787855770.15 total 94.23 last 16.29) - open/close 1730 (time 1787855780.21 total 104.29 last 20.68) - open/close 1965 (time 1787855790.23 total 114.31 last 23.45) - open/close 2212 (time 1787855800.38 total 124.45 last 24.35) - open/close 2473 (time 1787855810.99 total 135.06 last 24.59) - open/close 2649 (time 1787855821.10 total 145.18 last 17.40) - open/close 2890 (time 1787855831.81 total 155.88 last 22.52) - open/close 3109 (time 1787855842.34 total 166.42 last 20.78) - open/close 3310 (time 1787855852.46 total 176.53 last 19.88) - open/close 3553 (time 1787855863.08 total 187.15 last 22.88) - open/close 3783 (time 1787855873.10 total 197.18 last 22.94) - open/close 4036 (time 1787855884.01 total 208.09 last 23.20) - open/close 4295 (time 1787855894.83 total 218.91 last 23.93) - open/close 4509 (time 1787855904.85 total 228.93 last 21.36) - open/close 4774 (time 1787855914.86 total 238.94 last 26.48) - open/close 5030 (time 1787855924.87 total 248.94 last 25.59) - open/close 5272 (time 1787855935.18 total 259.25 last 23.47) - open/close 5515 (time 1787855946.02 total 270.10 last 22.41) - open/close 5690 (time 1787855956.98 total 281.05 last 15.98) - open/close 5869 (time 1787855967.73 total 291.81 last 16.64) - open/close 6008 (time 1787855978.68 total 302.75 last 12.70) - open/close 6105 (time 1787855988.78 total 312.85 last 9.60) - open/close 6353 (time 1787855999.41 total 323.49 last 23.32) - open/close 6561 (time 1787856010.05 total 334.12 last 19.56) - open/close 6806 (time 1787856020.39 total 344.47 last 23.68) - open/close 7056 (time 1787856030.42 total 354.50 last 24.92) - open/close 7268 (time 1787856040.43 total 364.50 last 21.20) - open/close 7488 (time 1787856050.60 total 374.68 last 21.62) - open/close 7699 (time 1787856060.61 total 384.68 last 21.09) - open/close 7963 (time 1787856070.62 total 394.70 last 26.36) - open/close 8135 (time 1787856080.88 total 404.95 last 16.77) - open/close 8406 (time 1787856090.89 total 414.96 last 27.07) - open/close 8658 (time 1787856101.73 total 425.81 last 23.23) - open/close 8902 (time 1787856112.75 total 436.83 last 22.15) - open/close 9097 (time 1787856122.76 total 446.84 last 19.48) - open/close 9322 (time 1787856133.16 total 457.24 last 21.64) - open/close 9521 (time 1787856143.30 total 467.37 last 19.63) - open/close 9780 (time 1787856154.11 total 478.18 last 23.96) - open/close 10000 (time 1787856163.36 total 487.43 last 23.78) - open/close 10236 (time 1787856173.36 total 497.43 last 23.59) - open/close 10473 (time 1787856183.36 total 507.44 last 23.69) - open/close 10753 (time 1787856193.83 total 517.91 last 26.74) - open/close 10969 (time 1787856203.84 total 527.91 last 21.59) - open/close 11161 (time 1787856214.58 total 538.65 last 17.88) - open/close 11384 (time 1787856224.97 total 549.05 last 21.45) - open/close 11619 (time 1787856235.56 total 559.63 last 22.20) - open/close 11859 (time 1787856245.81 total 569.88 last 23.42) - open/close 12025 (time 1787856255.82 total 579.90 last 16.58) - open/close 12133 (time 1787856265.84 total 589.92 last 10.78) - open/close 12308 (time 1787856276.75 total 600.83 last 16.04) - open/close 12488 (time 1787856286.77 total 610.84 last 17.97) - open/close 12691 (time 1787856296.79 total 620.86 last 20.26) - open/close 12939 (time 1787856306.81 total 630.88 last 24.75) - open/close 13133 (time 1787856316.81 total 640.88 last 19.40) - open/close 13346 (time 1787856329.51 total 653.59 last 16.76) - open/close 13583 (time 1787856339.53 total 663.60 last 23.66) - open/close 13704 (time 1787856350.39 total 674.46 last 11.14) - open/close 13898 (time 1787856360.39 total 684.46 last 19.40) - open/close 14110 (time 1787856370.39 total 694.47 last 21.20) - open/close 14320 (time 1787856380.39 total 704.47 last 21.00) - open/close 14510 (time 1787856390.47 total 714.55 last 18.85) - open/close 14689 (time 1787856401.20 total 725.27 last 16.69) - open/close 14917 (time 1787856411.23 total 735.30 last 22.73) - open/close 15106 (time 1787856421.49 total 745.56 last 18.42) - open/close 15220 (time 1787856431.61 total 755.69 last 11.26) - open/close 15403 (time 1787856442.42 total 766.49 last 16.94) - open/close 15636 (time 1787856452.44 total 776.51 last 23.26) - open/close 15819 (time 1787856463.35 total 787.42 last 16.77) - open/close 16074 (time 1787856473.80 total 797.87 last 24.40) - open/close 16286 (time 1787856484.67 total 808.74 last 19.51) - open/close 16518 (time 1787856494.90 total 818.98 last 22.66) - open/close 16777 (time 1787856505.58 total 829.66 last 24.25) - open/close 16921 (time 1787856515.88 total 839.95 last 13.99) - open/close 17054 (time 1787856526.33 total 850.40 last 12.73) - open/close 17325 (time 1787856539.01 total 863.08 last 21.38) - open/close 17456 (time 1787856549.01 total 873.08 last 13.10) - open/close 17686 (time 1787856559.03 total 883.10 last 22.95) - open/close 17910 (time 1787856569.03 total 893.11 last 22.40) - open/close 18125 (time 1787856579.03 total 903.11 last 21.49) - open/close 18358 (time 1787856589.43 total 913.51 last 22.40) - open/close 18596 (time 1787856599.44 total 923.52 last 23.78) - open/close 18857 (time 1787856609.46 total 933.53 last 26.06) - open/close 19434 (time 1787856619.47 total 943.54 last 57.62) - open/close 20000 (time 1787856628.50 total 952.58 last 62.65) - open/close 20666 (time 1787856638.51 total 962.58 last 66.56) - open/close 21293 (time 1787856648.51 total 972.59 last 62.69) - open/close 22003 (time 1787856658.52 total 982.60 last 70.91) - open/close 22636 (time 1787856668.53 total 992.60 last 63.28) - open/close 23303 (time 1787856678.54 total 1002.62 last 66.60) - open/close 23916 (time 1787856688.55 total 1012.62 last 61.27) - open/close 24513 (time 1787856698.59 total 1022.67 last 59.42) - open/close 25093 (time 1787856708.61 total 1032.69 last 57.90) - open/close 25662 (time 1787856718.62 total 1042.69 last 56.87) - open/close 26306 (time 1787856728.62 total 1052.70 last 64.36) - open/close 26924 (time 1787856738.65 total 1062.73 last 61.61) - open/close 27524 (time 1787856748.66 total 1072.73 last 59.98) - open/close 28135 (time 1787856758.69 total 1082.76 last 60.93) - open/close 28748 (time 1787856768.69 total 1092.77 last 61.25) - open/close 29340 (time 1787856778.70 total 1102.77 last 59.17) - open/close 29909 (time 1787856788.72 total 1112.80 last 56.76) - open/close 30537 (time 1787856798.74 total 1122.81 last 62.72) - open/close 31148 (time 1787856808.75 total 1132.82 last 61.04) - open/close 31708 (time 1787856818.75 total 1142.83 last 55.97) - open/close 32258 (time 1787856828.75 total 1152.83 last 54.99) - open/close 32803 (time 1787856838.76 total 1162.83 last 54.48) - open/close 33408 (time 1787856848.76 total 1172.84 last 60.48) - open/close 34052 (time 1787856858.76 total 1182.84 last 64.39) - open/close 34644 (time 1787856868.79 total 1192.87 last 59.03) - open/close 35271 (time 1787856878.80 total 1202.88 last 62.64) - open/close 35873 (time 1787856888.81 total 1212.89 last 60.12) - open/close 36485 (time 1787856898.82 total 1222.89 last 61.17) - open/close 37036 (time 1787856908.83 total 1232.90 last 55.04) - open/close 37587 (time 1787856918.83 total 1242.91 last 55.09) - open/close 38201 (time 1787856928.83 total 1252.91 last 61.39) - open/close 38815 (time 1787856938.84 total 1262.91 last 61.36) - open/close 39447 (time 1787856948.84 total 1272.92 last 63.19) - open/close 40000 (time 1787856958.09 total 1282.16 last 59.78) - open/close 40632 (time 1787856968.10 total 1292.18 last 63.12) - open/close 41239 (time 1787856978.11 total 1302.18 last 60.67) - open/close 41879 (time 1787856988.13 total 1312.21 last 63.85) - open/close 42502 (time 1787856998.17 total 1322.24 last 62.08) - open/close 43153 (time 1787857008.17 total 1332.24 last 65.07) - open/close 43766 (time 1787857018.17 total 1342.25 last 61.29) - open/close 44437 (time 1787857028.19 total 1352.26 last 67.01) - open/close 45101 (time 1787857038.19 total 1362.26 last 66.38) - open/close 45799 (time 1787857048.19 total 1372.27 last 69.76) - open/close 46401 (time 1787857058.20 total 1382.28 last 60.15) - open/close 47017 (time 1787857068.22 total 1392.30 last 61.46) - open/close 47666 (time 1787857078.23 total 1402.31 last 64.85) - open/close 48296 (time 1787857088.23 total 1412.31 last 62.99) - open/close 48888 (time 1787857098.25 total 1422.32 last 59.14) - open/close 49490 (time 1787857108.28 total 1432.35 last 60.02) total: 50000 open/close in 1440.45 seconds: 34.71 ops/second - unlinked 0 (time 1787857121 ; total 0 ; last 0) - unlinked 10000 (time 1787857291 ; total 170 ; last 170) - unlinked 20000 (time 1787857458 ; total 337 ; last 167) - unlinked 30000 (time 1787857621 ; total 500 ; last 163) - unlinked 40000 (time 1787857781 ; total 660 ; last 160) total: 50000 unlinks in 820 seconds: 60.975609 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local oleg326-server: modules unloaded. pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2