== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:10:11 (1787789411) start mds service on oleg315-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg315-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1787789498.21 total 10.01 last 20.29) - open/close 419 (time 1787789508.21 total 20.01 last 21.59) - open/close 624 (time 1787789519.11 total 30.91 last 18.81) - open/close 851 (time 1787789529.36 total 41.16 last 22.15) - open/close 1069 (time 1787789540.29 total 52.09 last 19.94) - open/close 1203 (time 1787789551.11 total 62.91 last 12.39) - open/close 1337 (time 1787789561.92 total 73.72 last 12.39) - open/close 1479 (time 1787789572.87 total 84.67 last 12.97) - open/close 1730 (time 1787789583.19 total 94.99 last 24.33) - open/close 1915 (time 1787789593.81 total 105.61 last 17.42) - open/close 2070 (time 1787789603.82 total 115.62 last 15.47) - open/close 2325 (time 1787789614.80 total 126.60 last 23.23) - open/close 2528 (time 1787789625.74 total 137.54 last 18.57) - open/close 2761 (time 1787789636.03 total 147.83 last 22.64) - open/close 2925 (time 1787789646.21 total 158.01 last 16.11) - open/close 3215 (time 1787789656.22 total 168.02 last 28.97) - open/close 3482 (time 1787789666.23 total 178.03 last 26.68) - open/close 3720 (time 1787789676.87 total 188.67 last 22.36) - open/close 3980 (time 1787789687.87 total 199.67 last 23.63) - open/close 4104 (time 1787789698.69 total 210.49 last 11.46) - open/close 4311 (time 1787789708.73 total 220.53 last 20.61) - open/close 4596 (time 1787789719.47 total 231.27 last 26.55) - open/close 4771 (time 1787789729.55 total 241.35 last 17.36) - open/close 4964 (time 1787789739.56 total 251.36 last 19.28) - open/close 5192 (time 1787789750.03 total 261.83 last 21.77) - open/close 5449 (time 1787789760.58 total 272.38 last 24.35) - open/close 5683 (time 1787789771.01 total 282.81 last 22.45) - open/close 5904 (time 1787789781.02 total 292.82 last 22.07) - open/close 6120 (time 1787789791.03 total 302.83 last 21.57) - open/close 6384 (time 1787789801.04 total 312.84 last 26.39) - open/close 6604 (time 1787789811.46 total 323.26 last 21.12) - open/close 6877 (time 1787789821.48 total 333.28 last 27.25) - open/close 7073 (time 1787789831.48 total 343.28 last 19.59) - open/close 7307 (time 1787789841.80 total 353.60 last 22.69) - open/close 7520 (time 1787789852.89 total 364.69 last 19.20) - open/close 7657 (time 1787789862.89 total 374.69 last 13.69) - open/close 7931 (time 1787789873.68 total 385.48 last 25.40) - open/close 8139 (time 1787789884.54 total 396.34 last 19.14) - open/close 8279 (time 1787789895.43 total 407.23 last 12.86) - open/close 8422 (time 1787789905.44 total 417.24 last 14.28) - open/close 8568 (time 1787789915.45 total 427.25 last 14.60) - open/close 8722 (time 1787789925.57 total 437.37 last 15.22) - open/close 8824 (time 1787789936.07 total 447.87 last 9.71) - open/close 9067 (time 1787789946.63 total 458.43 last 23.00) - open/close 9271 (time 1787789956.64 total 468.44 last 20.39) - open/close 9485 (time 1787789966.66 total 478.46 last 21.34) - open/close 9728 (time 1787789979.60 total 491.40 last 18.79) - open/close 9988 (time 1787789989.61 total 501.41 last 25.98) - open/close 10201 (time 1787790000.01 total 511.81 last 20.48) - open/close 10458 (time 1787790010.70 total 522.50 last 24.04) - open/close 10669 (time 1787790020.74 total 532.54 last 21.01) - open/close 10928 (time 1787790030.79 total 542.59 last 25.78) - open/close 11135 (time 1787790040.97 total 552.77 last 20.34) - open/close 11357 (time 1787790050.97 total 562.77 last 22.19) - open/close 11528 (time 1787790061.44 total 573.24 last 16.32) - open/close 11682 (time 1787790071.45 total 583.25 last 15.39) - open/close 11834 (time 1787790081.79 total 593.59 last 14.69) - open/close 11985 (time 1787790091.80 total 603.60 last 15.10) - open/close 12193 (time 1787790102.40 total 614.20 last 19.62) - open/close 12328 (time 1787790113.34 total 625.14 last 12.34) - open/close 12464 (time 1787790123.35 total 635.15 last 13.60) - open/close 12685 (time 1787790133.35 total 645.15 last 22.10) - open/close 12896 (time 1787790143.70 total 655.50 last 20.38) - open/close 13139 (time 1787790154.12 total 665.92 last 23.32) - open/close 13383 (time 1787790164.55 total 676.35 last 23.40) - open/close 13605 (time 1787790175.00 total 686.80 last 21.24) - open/close 13876 (time 1787790185.99 total 697.79 last 24.67) - open/close 14474 (time 1787790196.01 total 707.81 last 59.66) - open/close 15389 (time 1787790206.02 total 717.82 last 91.45) - open/close 16396 (time 1787790216.02 total 727.82 last 100.70) - open/close 17394 (time 1787790226.02 total 737.82 last 99.74) - open/close 18339 (time 1787790236.02 total 747.82 last 94.49) - open/close 19339 (time 1787790246.02 total 757.82 last 100.00) - open/close 20000 (time 1787790253.08 total 764.88 last 93.72) - open/close 20987 (time 1787790263.08 total 774.88 last 98.63) - open/close 21900 (time 1787790273.08 total 784.88 last 91.30) - open/close 22840 (time 1787790283.09 total 794.89 last 93.97) - open/close 23704 (time 1787790293.10 total 804.90 last 86.31) - open/close 24734 (time 1787790303.10 total 814.90 last 102.98) - open/close 25701 (time 1787790313.11 total 824.91 last 96.64) - open/close 26691 (time 1787790323.11 total 834.91 last 98.93) - open/close 27715 (time 1787790333.12 total 844.92 last 102.38) - open/close 28658 (time 1787790343.12 total 854.92 last 94.29) - open/close 29693 (time 1787790353.12 total 864.92 last 103.48) - open/close 30000 (time 1787790356.30 total 868.10 last 96.42) - open/close 30967 (time 1787790366.31 total 878.11 last 96.62) - open/close 31918 (time 1787790376.31 total 888.11 last 95.09) - open/close 32874 (time 1787790386.31 total 898.11 last 95.58) - open/close 33829 (time 1787790396.31 total 908.11 last 95.50) - open/close 34720 (time 1787790406.32 total 918.12 last 89.06) - open/close 35657 (time 1787790416.32 total 928.12 last 93.68) - open/close 36574 (time 1787790426.33 total 938.12 last 91.66) - open/close 37516 (time 1787790436.34 total 948.14 last 94.07) - open/close 38503 (time 1787790446.34 total 958.14 last 98.66) - open/close 39431 (time 1787790456.35 total 968.15 last 92.76) - open/close 40000 (time 1787790461.88 total 973.68 last 102.76) - open/close 40936 (time 1787790471.90 total 983.70 last 93.49) - open/close 41913 (time 1787790481.90 total 993.70 last 97.70) - open/close 42861 (time 1787790491.90 total 1003.70 last 94.76) - open/close 43754 (time 1787790501.90 total 1013.70 last 89.28) - open/close 44719 (time 1787790511.91 total 1023.71 last 96.46) - open/close 45677 (time 1787790521.91 total 1033.71 last 95.74) - open/close 46719 (time 1787790531.92 total 1043.72 last 104.18) - open/close 47660 (time 1787790541.93 total 1053.73 last 93.97) - open/close 48665 (time 1787790551.94 total 1063.74 last 100.40) - open/close 49562 (time 1787790561.94 total 1073.74 last 89.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1074.54 seconds: 46.19 ops/second - unlinked 0 (time 1787790566 ; total 0 ; last 0) - unlinked 10000 (time 1787790669 ; total 103 ; last 103) - unlinked 20000 (time 1787790738 ; total 172 ; last 69) - unlinked 30000 (time 1787790795 ; total 229 ; last 57) - unlinked 40000 (time 1787790848 ; total 282 ; last 53) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 342 seconds: 145.119888 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded. pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2