== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:00:55 (1786816855) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg302-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1786816958.31 total 10.34 last 13.44) - open/close 278 (time 1786816968.75 total 20.79 last 13.31) - open/close 403 (time 1786816978.77 total 30.81 last 12.47) - open/close 533 (time 1786816988.79 total 40.82 last 12.98) - open/close 701 (time 1786816999.20 total 51.24 last 16.13) - open/close 889 (time 1786817010.14 total 62.18 last 17.19) - open/close 1107 (time 1786817020.38 total 72.41 last 21.30) - open/close 1390 (time 1786817031.42 total 83.45 last 25.63) - open/close 1575 (time 1786817041.43 total 93.46 last 18.48) - open/close 1757 (time 1786817052.20 total 104.23 last 16.90) - open/close 1960 (time 1786817062.69 total 114.73 last 19.34) - open/close 2207 (time 1786817072.94 total 124.98 last 24.09) - open/close 2346 (time 1786817082.95 total 134.98 last 13.89) - open/close 2525 (time 1786817093.99 total 146.02 last 16.22) - open/close 2647 (time 1786817104.04 total 156.07 last 12.14) - open/close 2876 (time 1786817114.04 total 166.08 last 22.88) - open/close 3135 (time 1786817124.05 total 176.08 last 25.90) - open/close 3334 (time 1786817134.05 total 186.08 last 19.90) - open/close 3441 (time 1786817144.67 total 196.70 last 10.07) - open/close 3707 (time 1786817154.71 total 206.75 last 26.49) - open/close 3878 (time 1786817165.28 total 217.32 last 16.18) - open/close 4045 (time 1786817176.47 total 228.51 last 14.92) - open/close 4266 (time 1786817187.22 total 239.26 last 20.55) - open/close 4434 (time 1786817198.25 total 250.28 last 15.24) - open/close 4549 (time 1786817209.05 total 261.08 last 10.65) - open/close 4755 (time 1786817219.06 total 271.10 last 20.57) - open/close 4943 (time 1786817229.98 total 282.01 last 17.22) - open/close 5139 (time 1786817240.81 total 292.84 last 18.10) - open/close 5361 (time 1786817251.23 total 303.27 last 21.29) - open/close 5483 (time 1786817261.61 total 313.65 last 11.76) - open/close 5592 (time 1786817271.61 total 323.65 last 10.90) - open/close 5790 (time 1786817283.50 total 335.53 last 16.66) - open/close 6009 (time 1786817293.51 total 345.54 last 21.88) - open/close 6232 (time 1786817303.52 total 355.55 last 22.28) - open/close 6429 (time 1786817313.72 total 365.75 last 19.32) - open/close 6570 (time 1786817325.98 total 378.02 last 11.49) - open/close 6688 (time 1786817337.06 total 389.09 last 10.66) - open/close 6880 (time 1786817347.25 total 399.29 last 18.83) - open/close 7017 (time 1786817357.54 total 409.58 last 13.31) - open/close 7297 (time 1786817368.17 total 420.20 last 26.36) - open/close 7473 (time 1786817378.40 total 430.43 last 17.20) - open/close 7675 (time 1786817389.00 total 441.04 last 19.05) - open/close 7973 (time 1786817399.46 total 451.50 last 28.49) - open/close 8124 (time 1786817410.22 total 462.25 last 14.04) - open/close 8367 (time 1786817420.22 total 472.26 last 24.28) - open/close 8587 (time 1786817430.23 total 482.26 last 22.00) - open/close 8843 (time 1786817440.49 total 492.53 last 24.93) - open/close 9018 (time 1786817451.10 total 503.13 last 16.50) - open/close 9167 (time 1786817461.10 total 513.14 last 14.89) - open/close 9371 (time 1786817472.06 total 524.09 last 18.63) - open/close 9638 (time 1786817482.06 total 534.09 last 26.69) - open/close 9904 (time 1786817492.65 total 544.68 last 25.12) - open/close 10000 (time 1786817496.34 total 548.38 last 25.98) - open/close 10265 (time 1786817506.72 total 558.76 last 25.53) - open/close 10521 (time 1786817516.72 total 568.76 last 25.60) - open/close 10758 (time 1786817526.74 total 578.78 last 23.65) - open/close 11018 (time 1786817536.76 total 588.79 last 25.97) - open/close 11282 (time 1786817547.22 total 599.26 last 25.23) - open/close 11548 (time 1786817557.35 total 609.39 last 26.26) - open/close 11782 (time 1786817568.17 total 620.20 last 21.63) - open/close 12049 (time 1786817578.92 total 630.96 last 24.83) - open/close 12292 (time 1786817589.15 total 641.19 last 23.74) - open/close 12565 (time 1786817599.64 total 651.67 last 26.04) - open/close 12904 (time 1786817610.02 total 662.05 last 32.66) - open/close 13164 (time 1786817620.02 total 672.06 last 25.98) - open/close 13401 (time 1786817630.03 total 682.06 last 23.68) - open/close 13662 (time 1786817640.49 total 692.53 last 24.94) - open/close 13971 (time 1786817651.23 total 703.27 last 28.77) - open/close 14231 (time 1786817661.24 total 713.27 last 26.00) - open/close 14443 (time 1786817671.65 total 723.69 last 20.35) - open/close 14692 (time 1786817682.29 total 734.32 last 23.41) - open/close 14863 (time 1786817693.88 total 745.91 last 14.76) - open/close 15034 (time 1786817704.06 total 756.10 last 16.79) - open/close 15198 (time 1786817714.55 total 766.58 last 15.64) - open/close 15483 (time 1786817725.10 total 777.13 last 27.01) - open/close 15752 (time 1786817735.11 total 787.15 last 26.86) - open/close 15970 (time 1786817745.13 total 797.16 last 21.76) - open/close 16214 (time 1786817755.57 total 807.61 last 23.37) - open/close 16472 (time 1786817766.46 total 818.49 last 23.70) - open/close 16718 (time 1786817776.49 total 828.53 last 24.51) - open/close 16993 (time 1786817787.10 total 839.14 last 25.93) - open/close 17250 (time 1786817798.39 total 850.42 last 22.77) - open/close 17508 (time 1786817808.80 total 860.83 last 24.79) - open/close 17722 (time 1786817818.81 total 870.84 last 21.38) - open/close 17886 (time 1786817829.24 total 881.27 last 15.72) - open/close 18042 (time 1786817839.28 total 891.31 last 15.53) - open/close 18310 (time 1786817849.33 total 901.37 last 26.66) - open/close 18502 (time 1786817859.83 total 911.86 last 18.29) - open/close 18742 (time 1786817870.82 total 922.85 last 21.84) - open/close 18955 (time 1786817881.00 total 933.03 last 20.93) - open/close 19218 (time 1786817891.75 total 943.78 last 24.46) - open/close 19474 (time 1786817902.81 total 954.85 last 23.13) - open/close 19715 (time 1786817912.83 total 964.87 last 24.05) - open/close 19945 (time 1786817923.25 total 975.29 last 22.07) - open/close 20000 (time 1786817925.51 total 977.54 last 24.39) - open/close 20251 (time 1786817935.51 total 987.55 last 25.09) - open/close 21142 (time 1786817945.52 total 997.55 last 89.04) - open/close 22013 (time 1786817955.52 total 1007.55 last 87.09) - open/close 22887 (time 1786817965.53 total 1017.56 last 87.34) - open/close 23747 (time 1786817975.53 total 1027.56 last 85.97) - open/close 24579 (time 1786817985.53 total 1037.57 last 83.18) - open/close 25336 (time 1786817995.54 total 1047.57 last 75.67) - open/close 25851 (time 1786818005.58 total 1057.62 last 51.26) - open/close 26335 (time 1786818015.61 total 1067.64 last 48.27) - open/close 27151 (time 1786818025.61 total 1077.65 last 81.57) - open/close 28007 (time 1786818035.63 total 1087.66 last 85.45) - open/close 28808 (time 1786818045.64 total 1097.68 last 80.00) - open/close 29666 (time 1786818055.65 total 1107.68 last 85.75) - open/close 30000 (time 1786818059.02 total 1111.05 last 99.14) - open/close 31035 (time 1786818069.02 total 1121.06 last 103.45) - open/close 31959 (time 1786818079.02 total 1131.06 last 92.39) - open/close 32887 (time 1786818089.04 total 1141.08 last 92.59) - open/close 33842 (time 1786818099.06 total 1151.10 last 95.33) - open/close 34626 (time 1786818109.08 total 1161.12 last 78.23) - open/close 35306 (time 1786818119.10 total 1171.13 last 67.89) - open/close 36197 (time 1786818129.10 total 1181.14 last 89.07) - open/close 37004 (time 1786818139.11 total 1191.14 last 80.65) - open/close 37917 (time 1786818149.12 total 1201.15 last 91.23) - open/close 38697 (time 1786818159.12 total 1211.16 last 77.93) - open/close 39589 (time 1786818169.13 total 1221.16 last 89.17) - open/close 40000 (time 1786818174.03 total 1226.07 last 83.83) - open/close 40792 (time 1786818184.04 total 1236.07 last 79.14) - open/close 41560 (time 1786818194.04 total 1246.08 last 76.76) - open/close 42502 (time 1786818204.04 total 1256.08 last 94.19) - open/close 43421 (time 1786818214.05 total 1266.08 last 91.85) - open/close 44341 (time 1786818224.05 total 1276.09 last 91.96) - open/close 45275 (time 1786818234.06 total 1286.09 last 93.36) - open/close 46151 (time 1786818244.06 total 1296.09 last 87.58) - open/close 47118 (time 1786818254.07 total 1306.10 last 96.64) - open/close 48061 (time 1786818264.07 total 1316.11 last 94.25) - open/close 48903 (time 1786818274.08 total 1326.12 last 84.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1335.40 seconds: 37.17 ops/second - unlinked 0 (time 1786818288 ; total 0 ; last 0) - unlinked 10000 (time 1786818408 ; total 120 ; last 120) - unlinked 20000 (time 1786818526 ; total 238 ; last 118) - unlinked 30000 (time 1786818588 ; total 300 ; last 62) - unlinked 40000 (time 1786818659 ; total 371 ; last 71) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 438 seconds: 113.312782 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2