== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:42:09 (1786830129) start mds service on oleg356-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg356-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1786830235.64 total 10.01 last 14.69) - open/close 297 (time 1786830246.62 total 20.99 last 13.66) - open/close 500 (time 1786830256.63 total 31.00 last 20.28) - open/close 577 (time 1786830266.65 total 41.01 last 7.69) - open/close 670 (time 1786830276.65 total 51.02 last 9.30) - open/close 901 (time 1786830286.66 total 61.03 last 23.08) - open/close 1159 (time 1786830296.73 total 71.10 last 25.62) - open/close 1408 (time 1786830306.95 total 81.32 last 24.37) - open/close 1637 (time 1786830318.11 total 92.47 last 20.52) - open/close 1943 (time 1786830328.67 total 103.03 last 28.98) - open/close 2132 (time 1786830338.97 total 113.34 last 18.34) - open/close 2400 (time 1786830348.97 total 123.34 last 26.79) - open/close 2513 (time 1786830359.46 total 133.83 last 10.77) - open/close 2722 (time 1786830369.96 total 144.32 last 19.92) - open/close 3038 (time 1786830380.90 total 155.27 last 28.87) - open/close 3244 (time 1786830390.95 total 165.31 last 20.51) - open/close 3380 (time 1786830401.66 total 176.03 last 12.69) - open/close 3529 (time 1786830411.67 total 186.04 last 14.89) - open/close 3761 (time 1786830421.86 total 196.23 last 22.76) - open/close 4010 (time 1786830432.24 total 206.60 last 24.01) - open/close 4264 (time 1786830442.66 total 217.02 last 24.37) - open/close 4575 (time 1786830452.66 total 227.03 last 31.08) - open/close 4753 (time 1786830463.00 total 237.37 last 17.22) - open/close 5041 (time 1786830473.39 total 247.75 last 27.74) - open/close 5262 (time 1786830483.39 total 257.75 last 22.10) - open/close 5412 (time 1786830494.82 total 269.19 last 13.12) - open/close 5618 (time 1786830505.30 total 279.67 last 19.65) - open/close 5844 (time 1786830515.31 total 289.68 last 22.59) - open/close 6036 (time 1786830525.32 total 299.68 last 19.18) - open/close 6136 (time 1786830535.72 total 310.08 last 9.62) - open/close 6334 (time 1786830545.72 total 320.08 last 19.79) - open/close 6580 (time 1786830556.64 total 331.00 last 22.53) - open/close 6849 (time 1786830566.64 total 341.01 last 26.89) - open/close 7096 (time 1786830577.07 total 351.44 last 23.68) - open/close 7246 (time 1786830587.42 total 361.79 last 14.49) - open/close 7549 (time 1786830598.19 total 372.55 last 28.15) - open/close 7807 (time 1786830608.20 total 382.57 last 25.76) - open/close 7980 (time 1786830618.21 total 392.58 last 17.28) - open/close 8249 (time 1786830628.90 total 403.27 last 25.16) - open/close 8570 (time 1786830638.90 total 413.27 last 32.09) - open/close 8801 (time 1786830649.44 total 423.81 last 21.92) - open/close 9059 (time 1786830659.44 total 433.81 last 25.79) - open/close 9358 (time 1786830669.67 total 444.03 last 29.25) - open/close 9624 (time 1786830679.68 total 454.05 last 26.56) - open/close 9863 (time 1786830689.75 total 464.12 last 23.72) - open/close 10000 (time 1786830694.89 total 469.26 last 26.66) - open/close 10159 (time 1786830704.91 total 479.27 last 15.87) - open/close 10405 (time 1786830715.47 total 489.84 last 23.29) - open/close 10608 (time 1786830725.47 total 499.84 last 20.29) - open/close 10807 (time 1786830736.69 total 511.06 last 17.74) - open/close 10894 (time 1786830747.56 total 521.92 last 8.01) - open/close 11116 (time 1786830758.37 total 532.73 last 20.53) - open/close 11236 (time 1786830769.11 total 543.47 last 11.17) - open/close 11480 (time 1786830779.11 total 553.48 last 24.39) - open/close 11702 (time 1786830789.12 total 563.49 last 22.17) - open/close 11953 (time 1786830799.71 total 574.08 last 23.70) - open/close 12211 (time 1786830809.78 total 584.15 last 25.63) - open/close 12464 (time 1786830819.93 total 594.30 last 24.92) - open/close 12671 (time 1786830830.95 total 605.32 last 18.79) - open/close 12913 (time 1786830841.50 total 615.87 last 22.93) - open/close 13068 (time 1786830852.14 total 626.50 last 14.57) - open/close 13300 (time 1786830862.18 total 636.54 last 23.11) - open/close 13563 (time 1786830872.87 total 647.23 last 24.60) - open/close 13735 (time 1786830882.97 total 657.34 last 17.02) - open/close 13912 (time 1786830892.98 total 667.34 last 17.69) - open/close 14129 (time 1786830903.00 total 677.36 last 21.66) - open/close 14356 (time 1786830913.03 total 687.39 last 22.63) - open/close 14595 (time 1786830923.03 total 697.39 last 23.90) - open/close 14835 (time 1786830933.15 total 707.52 last 23.70) - open/close 15172 (time 1786830943.16 total 717.52 last 33.69) - open/close 15393 (time 1786830953.17 total 727.54 last 22.06) - open/close 15635 (time 1786830964.13 total 738.50 last 22.08) - open/close 15909 (time 1786830974.13 total 748.50 last 27.40) - open/close 16109 (time 1786830984.95 total 759.31 last 18.49) - open/close 16246 (time 1786830995.51 total 769.87 last 12.98) - open/close 16388 (time 1786831008.25 total 782.61 last 11.14) - open/close 16630 (time 1786831018.66 total 793.03 last 23.23) - open/close 16907 (time 1786831028.69 total 803.06 last 27.63) - open/close 17151 (time 1786831038.69 total 813.06 last 24.39) - open/close 17389 (time 1786831048.70 total 823.06 last 23.79) - open/close 17684 (time 1786831058.72 total 833.08 last 29.44) - open/close 17866 (time 1786831069.09 total 843.46 last 17.54) - open/close 18159 (time 1786831079.97 total 854.34 last 26.94) - open/close 18372 (time 1786831089.97 total 864.34 last 21.30) - open/close 18507 (time 1786831100.52 total 874.89 last 12.79) - open/close 18701 (time 1786831111.29 total 885.66 last 18.01) - open/close 18910 (time 1786831121.30 total 895.67 last 20.88) - open/close 19148 (time 1786831131.94 total 906.31 last 22.38) - open/close 19390 (time 1786831142.00 total 916.37 last 24.05) - open/close 19622 (time 1786831152.95 total 927.31 last 21.20) - open/close 19892 (time 1786831162.99 total 937.36 last 26.88) - open/close 20000 (time 1786831167.03 total 941.40 last 26.72) - open/close 20283 (time 1786831177.04 total 951.41 last 28.28) - open/close 20618 (time 1786831187.05 total 961.42 last 33.46) - open/close 21431 (time 1786831197.06 total 971.43 last 81.25) - open/close 22226 (time 1786831207.07 total 981.44 last 79.42) - open/close 22969 (time 1786831217.07 total 991.44 last 74.27) - open/close 23627 (time 1786831227.09 total 1001.45 last 65.71) - open/close 24158 (time 1786831237.09 total 1011.45 last 53.10) - open/close 24870 (time 1786831247.10 total 1021.47 last 71.12) - open/close 25693 (time 1786831257.10 total 1031.47 last 82.29) - open/close 26497 (time 1786831267.11 total 1041.47 last 80.34) - open/close 27344 (time 1786831277.12 total 1051.49 last 84.57) - open/close 28199 (time 1786831287.14 total 1061.50 last 85.39) - open/close 28912 (time 1786831297.14 total 1071.50 last 71.30) - open/close 29665 (time 1786831307.14 total 1081.51 last 75.27) - open/close 30000 (time 1786831310.64 total 1085.00 last 95.80) - open/close 30752 (time 1786831320.64 total 1095.01 last 75.17) - open/close 31562 (time 1786831330.67 total 1105.03 last 80.81) - open/close 32428 (time 1786831340.67 total 1115.04 last 86.56) - open/close 33391 (time 1786831350.68 total 1125.04 last 96.24) - open/close 34299 (time 1786831360.69 total 1135.05 last 90.69) - open/close 35153 (time 1786831370.69 total 1145.06 last 85.36) - open/close 35987 (time 1786831380.70 total 1155.06 last 83.36) - open/close 36931 (time 1786831390.70 total 1165.06 last 94.39) - open/close 37731 (time 1786831400.70 total 1175.06 last 80.00) - open/close 38450 (time 1786831410.72 total 1185.08 last 71.76) - open/close 39220 (time 1786831420.72 total 1195.09 last 76.95) - open/close 39960 (time 1786831430.73 total 1205.10 last 73.94) - open/close 40713 (time 1786831440.74 total 1215.10 last 75.27) - open/close 41545 (time 1786831450.75 total 1225.11 last 83.11) - open/close 42385 (time 1786831460.75 total 1235.12 last 83.95) - open/close 43194 (time 1786831470.76 total 1245.12 last 80.87) - open/close 43943 (time 1786831480.76 total 1255.13 last 74.86) - open/close 44750 (time 1786831490.77 total 1265.14 last 80.62) - open/close 45528 (time 1786831500.78 total 1275.15 last 77.74) - open/close 46220 (time 1786831510.78 total 1285.15 last 69.20) - open/close 47077 (time 1786831520.78 total 1295.15 last 85.67) - open/close 47878 (time 1786831530.79 total 1305.16 last 80.06) - open/close 48608 (time 1786831540.81 total 1315.17 last 72.87) - open/close 49339 (time 1786831550.81 total 1325.18 last 73.04) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1328.83 seconds: 37.35 ops/second - unlinked 0 (time 1786831558 ; total 0 ; last 0) - unlinked 10000 (time 1786831682 ; total 124 ; last 124) - unlinked 20000 (time 1786831806 ; total 248 ; last 124) - unlinked 30000 (time 1786831926 ; total 368 ; last 120) - unlinked 40000 (time 1786832045 ; total 487 ; last 119) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 610 seconds: 81.362297 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: modules unloaded. pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2