== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:36:07 (1787837767) start mds service on oleg441-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg441-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 78 (time 1787837864.42 total 10.66 last 7.32) - open/close 190 (time 1787837874.71 total 20.95 last 10.88) - open/close 287 (time 1787837885.58 total 31.82 last 8.92) - open/close 445 (time 1787837897.54 total 43.78 last 13.22) - open/close 606 (time 1787837908.48 total 54.72 last 14.71) - open/close 778 (time 1787837919.39 total 65.63 last 15.77) - open/close 852 (time 1787837929.40 total 75.64 last 7.39) - open/close 1112 (time 1787837939.67 total 85.91 last 25.32) - open/close 1215 (time 1787837949.70 total 95.94 last 10.26) - open/close 1334 (time 1787837960.60 total 106.84 last 10.92) - open/close 1432 (time 1787837970.64 total 116.88 last 9.76) - open/close 1563 (time 1787837980.95 total 127.19 last 12.71) - open/close 1661 (time 1787837991.57 total 137.81 last 9.23) - open/close 1919 (time 1787838001.59 total 147.83 last 25.75) - open/close 2032 (time 1787838013.86 total 160.10 last 9.20) - open/close 2184 (time 1787838023.87 total 170.11 last 15.19) - open/close 2279 (time 1787838034.87 total 181.11 last 8.64) - open/close 2368 (time 1787838045.99 total 192.22 last 8.01) - open/close 2597 (time 1787838056.11 total 202.35 last 22.62) - open/close 2763 (time 1787838066.20 total 212.44 last 16.44) - open/close 2888 (time 1787838076.21 total 222.45 last 12.49) - open/close 3054 (time 1787838086.34 total 232.58 last 16.39) - open/close 3222 (time 1787838097.37 total 243.61 last 15.23) - open/close 3323 (time 1787838109.87 total 256.11 last 8.08) - open/close 3461 (time 1787838120.29 total 266.53 last 13.25) - open/close 3639 (time 1787838130.44 total 276.68 last 17.53) - open/close 3727 (time 1787838140.62 total 286.86 last 8.65) - open/close 3791 (time 1787838151.62 total 297.86 last 5.82) - open/close 3872 (time 1787838162.05 total 308.29 last 7.77) - open/close 3932 (time 1787838173.01 total 319.25 last 5.48) - open/close 4057 (time 1787838184.03 total 330.27 last 11.34) - open/close 4140 (time 1787838194.65 total 340.89 last 7.82) - open/close 4256 (time 1787838204.87 total 351.11 last 11.34) - open/close 4367 (time 1787838215.83 total 362.07 last 10.13) - open/close 4552 (time 1787838225.84 total 372.08 last 18.50) - open/close 4674 (time 1787838235.88 total 382.12 last 12.15) - open/close 4789 (time 1787838246.02 total 392.26 last 11.33) - open/close 4906 (time 1787838257.03 total 403.27 last 10.63) - open/close 5058 (time 1787838267.67 total 413.91 last 14.30) - open/close 5109 (time 1787838278.48 total 424.72 last 4.72) - open/close 5202 (time 1787838288.98 total 435.22 last 8.85) - open/close 5346 (time 1787838300.42 total 446.66 last 12.59) - open/close 5411 (time 1787838311.44 total 457.68 last 5.90) - open/close 5451 (time 1787838321.94 total 468.18 last 3.81) - open/close 5572 (time 1787838331.95 total 478.19 last 12.10) - open/close 5743 (time 1787838342.87 total 489.11 last 15.65) - open/close 5916 (time 1787838352.92 total 499.16 last 17.21) - open/close 6116 (time 1787838363.61 total 509.85 last 18.71) - open/close 6292 (time 1787838373.63 total 519.87 last 17.58) - open/close 6444 (time 1787838383.95 total 530.19 last 14.72) - open/close 6665 (time 1787838394.64 total 540.88 last 20.68) - open/close 6835 (time 1787838404.88 total 551.12 last 16.61) - open/close 7022 (time 1787838415.94 total 562.18 last 16.90) - open/close 7166 (time 1787838426.14 total 572.38 last 14.13) - open/close 7195 (time 1787838436.26 total 582.50 last 2.86) - open/close 7411 (time 1787838446.82 total 593.06 last 20.46) - open/close 7642 (time 1787838456.84 total 603.08 last 23.06) - open/close 7830 (time 1787838466.84 total 613.08 last 18.79) - open/close 8046 (time 1787838476.87 total 623.11 last 21.55) - open/close 8118 (time 1787838487.07 total 633.31 last 7.06) - open/close 8257 (time 1787838497.49 total 643.73 last 13.33) - open/close 8440 (time 1787838507.60 total 653.84 last 18.11) - open/close 8674 (time 1787838518.43 total 664.67 last 21.60) - open/close 8839 (time 1787838529.10 total 675.34 last 15.46) - open/close 9059 (time 1787838539.12 total 685.36 last 21.97) - open/close 9182 (time 1787838550.15 total 696.39 last 11.15) - open/close 9288 (time 1787838560.91 total 707.15 last 9.85) - open/close 9378 (time 1787838570.96 total 717.20 last 8.96) - open/close 9539 (time 1787838581.92 total 728.16 last 14.68) - open/close 9663 (time 1787838592.16 total 738.40 last 12.12) - open/close 9861 (time 1787838602.95 total 749.19 last 18.35) - open/close 9987 (time 1787838613.46 total 759.70 last 11.99) - open/close 10065 (time 1787838624.09 total 770.33 last 7.34) - open/close 10216 (time 1787838634.32 total 780.56 last 14.75) - open/close 10416 (time 1787838644.44 total 790.68 last 19.76) - open/close 10588 (time 1787838655.57 total 801.81 last 15.46) - open/close 10756 (time 1787838666.37 total 812.61 last 15.55) - open/close 10954 (time 1787838676.94 total 823.18 last 18.73) - open/close 11025 (time 1787838687.20 total 833.44 last 6.92) - open/close 11188 (time 1787838697.20 total 843.44 last 16.29) - open/close 11352 (time 1787838708.11 total 854.35 last 15.04) - open/close 11499 (time 1787838718.12 total 864.36 last 14.69) - open/close 11660 (time 1787838728.21 total 874.45 last 15.95) - open/close 11771 (time 1787838738.21 total 884.45 last 11.10) - open/close 11937 (time 1787838748.63 total 894.87 last 15.93) - open/close 12060 (time 1787838758.81 total 905.05 last 12.08) - open/close 12222 (time 1787838768.86 total 915.10 last 16.12) - open/close 12374 (time 1787838778.87 total 925.11 last 15.19) - open/close 12532 (time 1787838789.33 total 935.57 last 15.10) - open/close 12700 (time 1787838799.34 total 945.58 last 16.78) - open/close 12818 (time 1787838809.72 total 955.96 last 11.37) - open/close 12910 (time 1787838819.74 total 965.98 last 9.18) - open/close 13076 (time 1787838830.29 total 976.53 last 15.74) - open/close 13166 (time 1787838841.54 total 987.78 last 8.00) - open/close 13353 (time 1787838851.87 total 998.11 last 18.10) - open/close 13466 (time 1787838864.73 total 1010.97 last 8.78) - open/close 13554 (time 1787838875.22 total 1021.46 last 8.39) - open/close 13755 (time 1787838885.23 total 1031.47 last 20.07) - open/close 13977 (time 1787838895.93 total 1042.17 last 20.76) - open/close 14144 (time 1787838906.32 total 1052.56 last 16.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14195) error: No space left on device total: 14195 open/close in 1054.78 seconds: 13.46 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76080) - No such process - unlinked 0 (time 1787838918 ; total 0 ; last 0) - unlinked 10000 (time 1787839071 ; total 153 ; last 153) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14195) error: No such file or directory total: 14195 unlinks in 199 seconds: 71.331657 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2