== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:10:16 (1787371816) Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg333-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server oleg333-server: oleg333-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg333-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 622 (time 1787372026.85 total 10.02 last 62.09) - open/close 1325 (time 1787372036.86 total 20.03 last 70.22) - open/close 2097 (time 1787372046.88 total 30.04 last 77.08) - open/close 2846 (time 1787372056.89 total 40.06 last 74.81) - open/close 3496 (time 1787372066.89 total 50.06 last 64.98) - open/close 4132 (time 1787372076.90 total 60.07 last 63.55) total: 4500 open/close in 65.81 seconds: 68.38 ops/second - /unlink 1212 (time 1787372097.15 total 10.00 last 121.17) - /unlink 2546 (time 1787372107.15 total 20.00 last 133.39) - /unlink 3842 (time 1787372117.16 total 30.01 last 129.56) total: 4500 /unlink in 35.56 seconds: 126.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 664 (time 1787372137.08 total 10.02 last 66.27) - open/close 1371 (time 1787372147.10 total 20.04 last 70.54) - open/close 1995 (time 1787372157.11 total 30.05 last 62.35) - open/close 2704 (time 1787372167.13 total 40.07 last 70.74) - open/close 3450 (time 1787372177.14 total 50.09 last 74.50) - open/close 4159 (time 1787372187.15 total 60.09 last 70.87) total: 4500 open/close in 65.68 seconds: 68.52 ops/second - /unlink 1172 (time 1787372207.76 total 10.00 last 117.18) - /unlink 2465 (time 1787372217.76 total 20.00 last 129.27) - /unlink 3817 (time 1787372227.77 total 30.01 last 135.17) total: 4500 /unlink in 35.36 seconds: 127.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 657 (time 1787372247.83 total 10.02 last 65.60) - open/close 1391 (time 1787372257.83 total 20.02 last 73.40) - open/close 1987 (time 1787372267.84 total 30.02 last 59.56) - open/close 2668 (time 1787372277.84 total 40.02 last 68.09) - open/close 3392 (time 1787372287.86 total 50.04 last 72.28) - open/close 4053 (time 1787372297.88 total 60.06 last 65.97) total: 4500 open/close in 66.91 seconds: 67.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1296 (time 1787372319.12 total 10.00 last 129.55) - /unlink 2617 (time 1787372329.12 total 20.00 last 132.09) - /unlink 3758 (time 1787372339.13 total 30.01 last 114.05) total: 4500 /unlink in 36.28 seconds: 124.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 592 (time 1787372360.10 total 10.00 last 59.18) - open/close 1310 (time 1787372370.10 total 20.00 last 71.79) - open/close 1964 (time 1787372380.11 total 30.01 last 65.36) - open/close 2560 (time 1787372390.11 total 40.01 last 59.59) - open/close 3218 (time 1787372400.12 total 50.02 last 65.78) - open/close 3939 (time 1787372410.15 total 60.05 last 71.88) total: 4500 open/close in 67.58 seconds: 66.59 ops/second - /unlink 1346 (time 1787372432.27 total 10.01 last 134.52) - /unlink 2709 (time 1787372442.27 total 20.01 last 136.27) - /unlink 4217 (time 1787372452.27 total 30.01 last 150.73) total: 4500 /unlink in 32.07 seconds: 140.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 631 (time 1787372469.27 total 10.00 last 63.07) - open/close 1380 (time 1787372479.29 total 20.02 last 74.79) - open/close 2076 (time 1787372489.32 total 30.05 last 69.36) - open/close 2674 (time 1787372499.32 total 40.06 last 59.79) - open/close 3457 (time 1787372509.33 total 50.06 last 78.24) - open/close 4160 (time 1787372519.35 total 60.08 last 70.15) total: 4500 open/close in 64.49 seconds: 69.78 ops/second - /unlink 1366 (time 1787372538.25 total 10.01 last 136.52) - /unlink 2577 (time 1787372548.26 total 20.01 last 121.05) - /unlink 3892 (time 1787372558.26 total 30.01 last 131.45) total: 4500 /unlink in 34.81 seconds: 129.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 689 (time 1787372577.40 total 10.01 last 68.87) - open/close 1322 (time 1787372587.41 total 20.01 last 63.24) - open/close 1984 (time 1787372597.41 total 30.02 last 66.16) - open/close 2724 (time 1787372607.42 total 40.03 last 73.96) - open/close 3385 (time 1787372617.43 total 50.03 last 66.04) - open/close 4160 (time 1787372627.43 total 60.04 last 77.45) total: 4500 open/close in 65.15 seconds: 69.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1202 (time 1787372647.00 total 10.00 last 120.15) - /unlink 2390 (time 1787372657.00 total 20.01 last 118.74) - /unlink 3536 (time 1787372667.01 total 30.02 last 114.53) total: 4500 /unlink in 36.73 seconds: 122.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 672 (time 1787372688.38 total 10.01 last 67.17) - open/close 1399 (time 1787372698.39 total 20.01 last 72.65) - open/close 2175 (time 1787372708.40 total 30.02 last 77.57) - open/close 2942 (time 1787372718.41 total 40.03 last 76.59) - open/close 3656 (time 1787372728.41 total 50.03 last 71.39) - open/close 4381 (time 1787372738.42 total 60.04 last 72.42) total: 4500 open/close in 61.66 seconds: 72.98 ops/second - /unlink 1223 (time 1787372753.85 total 10.00 last 122.28) - /unlink 2520 (time 1787372763.85 total 20.00 last 129.65) - /unlink 3909 (time 1787372773.86 total 30.01 last 138.83) total: 4500 /unlink in 34.71 seconds: 129.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 715 (time 1787372793.37 total 10.01 last 71.44) - open/close 1440 (time 1787372803.39 total 20.03 last 72.37) - open/close 2190 (time 1787372813.40 total 30.03 last 74.94) - open/close 2923 (time 1787372823.41 total 40.04 last 73.24) - open/close 3663 (time 1787372833.41 total 50.04 last 73.99) - open/close 4350 (time 1787372843.42 total 60.06 last 68.62) total: 4500 open/close in 62.16 seconds: 72.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1422 (time 1787372859.56 total 10.00 last 142.16) - /unlink 2703 (time 1787372869.57 total 20.01 last 128.02) - /unlink 4057 (time 1787372879.57 total 30.01 last 135.36) total: 4500 /unlink in 33.02 seconds: 136.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 632 (time 1787372897.78 total 10.01 last 63.15) - open/close 1098 (time 1787372907.79 total 20.02 last 46.56) - open/close 1686 (time 1787372917.79 total 30.02 last 58.78) - open/close 2393 (time 1787372927.80 total 40.03 last 70.63) - open/close 3126 (time 1787372937.80 total 50.03 last 73.27) - open/close 3837 (time 1787372947.81 total 60.03 last 71.09) total: 4500 open/close in 69.22 seconds: 65.01 ops/second - /unlink 1370 (time 1787372970.36 total 10.00 last 136.94) - /unlink 2686 (time 1787372980.36 total 20.01 last 131.57) - /unlink 4030 (time 1787372990.36 total 30.01 last 134.37) total: 4500 /unlink in 33.65 seconds: 133.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 638 (time 1787373008.82 total 10.01 last 63.75) - open/close 1298 (time 1787373018.83 total 20.02 last 65.94) - open/close 2013 (time 1787373028.84 total 30.03 last 71.43) - open/close 2723 (time 1787373038.86 total 40.05 last 70.87) - open/close 3440 (time 1787373048.86 total 50.05 last 71.69) - open/close 4129 (time 1787373058.87 total 60.05 last 68.85) total: 4500 open/close in 64.83 seconds: 69.41 ops/second - /unlink 1281 (time 1787373078.28 total 10.01 last 127.98) - /unlink 2463 (time 1787373088.28 total 20.01 last 118.16) - /unlink 3642 (time 1787373098.29 total 30.02 last 117.82) total: 4500 /unlink in 36.32 seconds: 123.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 703 (time 1787373118.53 total 10.00 last 70.27) - open/close 1330 (time 1787373128.54 total 20.01 last 62.68) - open/close 2049 (time 1787373138.54 total 30.01 last 71.89) - open/close 2709 (time 1787373148.54 total 40.01 last 65.99) - open/close 3432 (time 1787373158.55 total 50.02 last 72.26) - open/close 4116 (time 1787373168.55 total 60.02 last 68.40) total: 4500 open/close in 65.29 seconds: 68.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1366 (time 1787373188.11 total 10.00 last 136.58) - /unlink 2719 (time 1787373198.11 total 20.00 last 135.29) - /unlink 4004 (time 1787373208.11 total 30.00 last 128.49) total: 4500 /unlink in 33.68 seconds: 133.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 788 (time 1787373226.61 total 10.01 last 78.74) - open/close 1454 (time 1787373236.64 total 20.03 last 66.43) - open/close 2101 (time 1787373246.65 total 30.04 last 64.66) - open/close 2819 (time 1787373256.65 total 40.04 last 71.79) - open/close 3532 (time 1787373266.66 total 50.05 last 71.24) - open/close 4207 (time 1787373276.66 total 60.05 last 67.49) total: 4500 open/close in 64.60 seconds: 69.66 ops/second - /unlink 1377 (time 1787373295.26 total 10.01 last 137.61) - /unlink 2727 (time 1787373305.26 total 20.01 last 134.97) - /unlink 4041 (time 1787373315.26 total 30.01 last 131.34) total: 4500 /unlink in 33.99 seconds: 132.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 639 (time 1787373334.67 total 10.00 last 63.89) - open/close 1302 (time 1787373344.68 total 20.01 last 66.25) - open/close 2000 (time 1787373354.69 total 30.02 last 69.72) - open/close 2645 (time 1787373364.69 total 40.02 last 64.49) - open/close 3321 (time 1787373374.71 total 50.04 last 67.49) - open/close 4041 (time 1787373384.71 total 60.04 last 71.99) total: 4500 open/close in 67.05 seconds: 67.12 ops/second - /unlink 1277 (time 1787373406.08 total 10.00 last 127.64) - /unlink 2550 (time 1787373416.09 total 20.01 last 127.24) - /unlink 3876 (time 1787373426.09 total 30.01 last 132.59) total: 4500 /unlink in 34.59 seconds: 130.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 708 (time 1787373445.77 total 10.01 last 70.75) - open/close 1406 (time 1787373455.77 total 20.01 last 69.77) - open/close 2095 (time 1787373465.78 total 30.02 last 68.85) - open/close 2771 (time 1787373475.78 total 40.02 last 67.60) - open/close 3458 (time 1787373485.80 total 50.04 last 68.54) - open/close 4160 (time 1787373495.81 total 60.05 last 70.14) total: 4500 open/close in 64.68 seconds: 69.58 ops/second - /unlink 1225 (time 1787373516.45 total 10.00 last 122.49) - /unlink 2442 (time 1787373526.46 total 20.01 last 121.61) - /unlink 3659 (time 1787373536.47 total 30.02 last 121.58) total: 4500 /unlink in 36.81 seconds: 122.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 653 (time 1787373557.65 total 10.01 last 65.25) - open/close 1300 (time 1787373567.67 total 20.03 last 64.57) - open/close 1972 (time 1787373577.67 total 30.03 last 67.16) - open/close 2636 (time 1787373587.68 total 40.04 last 66.36) - open/close 3347 (time 1787373597.69 total 50.04 last 71.07) - open/close 4075 (time 1787373607.70 total 60.05 last 72.73) total: 4500 open/close in 66.40 seconds: 67.78 ops/second - /unlink 1322 (time 1787373627.96 total 10.00 last 132.17) - /unlink 2609 (time 1787373637.96 total 20.00 last 128.66) - /unlink 4027 (time 1787373647.97 total 30.01 last 141.66) total: 4500 /unlink in 33.68 seconds: 133.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 751 (time 1787373666.29 total 10.00 last 75.08) - open/close 1504 (time 1787373676.30 total 20.01 last 75.24) - open/close 2238 (time 1787373686.30 total 30.01 last 73.40) - open/close 2983 (time 1787373696.31 total 40.02 last 74.41) - open/close 3717 (time 1787373706.31 total 50.02 last 73.39) - open/close 4422 (time 1787373716.32 total 60.03 last 70.46) total: 4500 open/close in 61.11 seconds: 73.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1370 (time 1787373731.65 total 10.00 last 136.97) - /unlink 2594 (time 1787373741.65 total 20.00 last 122.38) - /unlink 3943 (time 1787373751.66 total 30.01 last 134.83) total: 4500 /unlink in 34.62 seconds: 129.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 625 (time 1787373771.02 total 10.01 last 62.44) - open/close 1262 (time 1787373781.05 total 20.04 last 63.52) - open/close 1896 (time 1787373791.08 total 30.06 last 63.25) - open/close 2581 (time 1787373801.09 total 40.07 last 68.42) - open/close 3354 (time 1787373811.09 total 50.08 last 77.27) - open/close 4093 (time 1787373821.10 total 60.08 last 73.85) total: 4500 open/close in 66.08 seconds: 68.10 ops/second - /unlink 1337 (time 1787373841.01 total 10.00 last 133.69) - /unlink 2541 (time 1787373851.02 total 20.01 last 120.31) - /unlink 3821 (time 1787373861.03 total 30.01 last 127.93) total: 4500 /unlink in 35.60 seconds: 126.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 756 (time 1787373881.36 total 10.00 last 75.58) - open/close 1423 (time 1787373891.38 total 20.02 last 66.62) - open/close 2005 (time 1787373901.38 total 30.02 last 58.16) - open/close 2695 (time 1787373911.38 total 40.02 last 69.00) - open/close 3405 (time 1787373921.39 total 50.03 last 70.98) - open/close 4109 (time 1787373931.40 total 60.04 last 70.34) total: 4500 open/close in 66.05 seconds: 68.13 ops/second - /unlink 1268 (time 1787373951.97 total 10.01 last 126.66) - /unlink 2525 (time 1787373961.98 total 20.02 last 125.65) - /unlink 3864 (time 1787373971.98 total 30.02 last 133.88) total: 4500 /unlink in 35.08 seconds: 128.29 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95343 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local oleg333-server: modules unloaded.