== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:21:18 (1782613278) Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg460-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg460-server oleg460-server: oleg460-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg460-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg460-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 615 (time 1782613499.51 total 10.01 last 61.43) - open/close 1285 (time 1782613509.51 total 20.01 last 66.98) - open/close 1928 (time 1782613519.52 total 30.02 last 64.26) - open/close 2614 (time 1782613529.54 total 40.04 last 68.47) - open/close 3228 (time 1782613539.55 total 50.05 last 61.34) - open/close 3883 (time 1782613549.56 total 60.06 last 65.41) - open/close 4493 (time 1782613559.58 total 70.08 last 60.88) total: 4500 open/close in 70.22 seconds: 64.09 ops/second - /unlink 1136 (time 1782613575.06 total 10.00 last 113.55) - /unlink 2237 (time 1782613585.07 total 20.02 last 109.94) - /unlink 3430 (time 1782613595.08 total 30.03 last 119.23) total: 4500 /unlink in 39.39 seconds: 114.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 614 (time 1782613619.35 total 10.02 last 61.28) - open/close 1175 (time 1782613629.36 total 20.03 last 56.05) - open/close 1876 (time 1782613639.36 total 30.03 last 70.08) - open/close 2498 (time 1782613649.36 total 40.03 last 62.19) - open/close 3148 (time 1782613659.37 total 50.04 last 64.94) - open/close 3791 (time 1782613669.38 total 60.05 last 64.23) - open/close 4437 (time 1782613679.41 total 70.08 last 64.44) total: 4500 open/close in 70.96 seconds: 63.41 ops/second - /unlink 1251 (time 1782613695.34 total 10.01 last 124.97) - /unlink 2387 (time 1782613705.34 total 20.01 last 113.60) - /unlink 3507 (time 1782613715.35 total 30.01 last 111.98) total: 4500 /unlink in 39.95 seconds: 112.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 644 (time 1782613741.94 total 10.02 last 64.30) - open/close 1194 (time 1782613751.94 total 20.02 last 54.99) - open/close 1809 (time 1782613761.95 total 30.03 last 61.43) - open/close 2362 (time 1782613771.97 total 40.05 last 55.17) - open/close 2971 (time 1782613781.99 total 50.07 last 60.78) - open/close 3509 (time 1782613792.00 total 60.08 last 53.79) - open/close 4054 (time 1782613802.00 total 70.08 last 54.45) total: 4500 open/close in 77.81 seconds: 57.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1202 (time 1782613825.01 total 10.00 last 120.17) - /unlink 2382 (time 1782613835.01 total 20.01 last 117.97) - /unlink 3498 (time 1782613845.01 total 30.01 last 111.60) total: 4500 /unlink in 39.00 seconds: 115.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 642 (time 1782613868.61 total 10.01 last 64.14) - open/close 1233 (time 1782613878.63 total 20.02 last 59.02) - open/close 1730 (time 1782613888.63 total 30.03 last 49.67) - open/close 2408 (time 1782613898.63 total 40.03 last 67.80) - open/close 3022 (time 1782613908.65 total 50.05 last 61.30) - open/close 3478 (time 1782613918.65 total 60.05 last 45.58) - open/close 3920 (time 1782613928.67 total 70.06 last 44.15) - open/close 4425 (time 1782613938.67 total 80.07 last 50.48) total: 4500 open/close in 81.43 seconds: 55.26 ops/second - /unlink 1163 (time 1782613956.27 total 10.00 last 116.26) - /unlink 2197 (time 1782613966.27 total 20.00 last 103.39) - /unlink 3323 (time 1782613976.28 total 30.01 last 112.53) - /unlink 4463 (time 1782613986.28 total 40.01 last 113.98) total: 4500 /unlink in 40.47 seconds: 111.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 651 (time 1782614000.92 total 10.01 last 65.02) - open/close 1289 (time 1782614010.93 total 20.02 last 63.74) - open/close 1904 (time 1782614020.94 total 30.03 last 61.45) - open/close 2490 (time 1782614030.94 total 40.03 last 58.57) - open/close 3117 (time 1782614040.95 total 50.03 last 62.70) - open/close 3812 (time 1782614050.95 total 60.04 last 69.48) - open/close 4433 (time 1782614060.95 total 70.04 last 62.06) total: 4500 open/close in 71.17 seconds: 63.23 ops/second - /unlink 1073 (time 1782614077.04 total 10.01 last 107.15) - /unlink 2300 (time 1782614087.04 total 20.02 last 122.61) - /unlink 3422 (time 1782614097.05 total 30.02 last 112.17) total: 4500 /unlink in 39.61 seconds: 113.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 661 (time 1782614121.54 total 10.01 last 66.01) - open/close 1260 (time 1782614131.55 total 20.02 last 59.84) - open/close 1902 (time 1782614141.56 total 30.04 last 64.11) - open/close 2433 (time 1782614151.59 total 40.06 last 52.96) - open/close 2984 (time 1782614161.60 total 50.08 last 55.03) - open/close 3637 (time 1782614171.61 total 60.09 last 65.22) - open/close 4238 (time 1782614181.62 total 70.10 last 60.06) total: 4500 open/close in 74.91 seconds: 60.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1203 (time 1782614201.68 total 10.00 last 120.28) - /unlink 2339 (time 1782614211.68 total 20.00 last 113.57) - /unlink 3426 (time 1782614221.68 total 30.01 last 108.66) total: 4500 /unlink in 39.22 seconds: 114.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 659 (time 1782614246.27 total 10.01 last 65.82) - open/close 1325 (time 1782614256.29 total 20.03 last 66.48) - open/close 1954 (time 1782614266.30 total 30.04 last 62.83) - open/close 2561 (time 1782614276.32 total 40.06 last 60.59) - open/close 3165 (time 1782614286.33 total 50.07 last 60.33) - open/close 3788 (time 1782614296.34 total 60.08 last 62.21) - open/close 4337 (time 1782614306.34 total 70.08 last 54.90) total: 4500 open/close in 72.76 seconds: 61.85 ops/second - /unlink 1176 (time 1782614323.61 total 10.01 last 117.52) - /unlink 2381 (time 1782614333.62 total 20.01 last 120.45) - /unlink 3544 (time 1782614343.62 total 30.02 last 116.25) total: 4500 /unlink in 37.91 seconds: 118.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 601 (time 1782614367.00 total 10.00 last 60.09) - open/close 1209 (time 1782614377.00 total 20.01 last 60.77) - open/close 1812 (time 1782614387.01 total 30.01 last 60.26) - open/close 2337 (time 1782614397.01 total 40.01 last 52.49) - open/close 2905 (time 1782614407.04 total 50.05 last 56.62) - open/close 3530 (time 1782614417.05 total 60.05 last 62.48) - open/close 4068 (time 1782614427.06 total 70.06 last 53.75) total: 4500 open/close in 76.79 seconds: 58.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1128 (time 1782614448.01 total 10.02 last 112.59) - /unlink 2287 (time 1782614458.02 total 20.02 last 115.83) - /unlink 3305 (time 1782614468.02 total 30.03 last 101.78) - /unlink 4313 (time 1782614478.03 total 40.03 last 100.73) total: 4500 /unlink in 41.48 seconds: 108.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 546 (time 1782614495.11 total 10.01 last 54.57) - open/close 1175 (time 1782614505.13 total 20.02 last 62.80) - open/close 1755 (time 1782614515.13 total 30.02 last 58.00) - open/close 2341 (time 1782614525.14 total 40.03 last 58.54) - open/close 2944 (time 1782614535.14 total 50.03 last 60.30) - open/close 3486 (time 1782614545.17 total 60.06 last 54.06) - open/close 4002 (time 1782614555.17 total 70.07 last 51.57) total: 4500 open/close in 78.84 seconds: 57.08 ops/second - /unlink 1129 (time 1782614579.68 total 10.00 last 112.89) - /unlink 2292 (time 1782614589.68 total 20.00 last 116.29) - /unlink 3325 (time 1782614599.69 total 30.01 last 103.25) - /unlink 4433 (time 1782614609.69 total 40.01 last 110.80) total: 4500 /unlink in 40.85 seconds: 110.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 617 (time 1782614626.36 total 10.00 last 61.67) - open/close 1245 (time 1782614636.38 total 20.02 last 62.71) - open/close 1805 (time 1782614646.39 total 30.03 last 55.94) - open/close 2349 (time 1782614656.41 total 40.05 last 54.29) - open/close 2996 (time 1782614666.41 total 50.06 last 64.66) - open/close 3630 (time 1782614676.43 total 60.07 last 63.29) - open/close 4215 (time 1782614686.43 total 70.07 last 58.48) total: 4500 open/close in 74.56 seconds: 60.35 ops/second - /unlink 1073 (time 1782614705.70 total 10.00 last 107.28) - /unlink 2135 (time 1782614715.70 total 20.01 last 106.13) - /unlink 3076 (time 1782614725.71 total 30.01 last 94.05) - /unlink 4232 (time 1782614735.71 total 40.02 last 115.54) total: 4500 /unlink in 42.85 seconds: 105.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 623 (time 1782614754.05 total 10.01 last 62.27) - open/close 1252 (time 1782614764.08 total 20.03 last 62.73) - open/close 1904 (time 1782614774.08 total 30.04 last 65.16) - open/close 2553 (time 1782614784.08 total 40.04 last 64.88) - open/close 3111 (time 1782614794.09 total 50.05 last 55.77) - open/close 3697 (time 1782614804.09 total 60.05 last 58.60) - open/close 4332 (time 1782614814.10 total 70.05 last 63.47) total: 4500 open/close in 73.11 seconds: 61.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 960 (time 1782614832.83 total 10.00 last 95.99) - /unlink 1907 (time 1782614842.84 total 20.01 last 94.60) - /unlink 2838 (time 1782614852.85 total 30.02 last 93.05) - /unlink 3911 (time 1782614862.85 total 40.02 last 107.29) total: 4500 /unlink in 45.06 seconds: 99.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 480 (time 1782614882.65 total 10.02 last 47.89) - open/close 915 (time 1782614892.67 total 20.05 last 43.38) - open/close 1400 (time 1782614902.69 total 30.06 last 48.44) - open/close 1870 (time 1782614912.69 total 40.06 last 46.99) - open/close 2373 (time 1782614922.70 total 50.07 last 50.27) - open/close 2863 (time 1782614932.72 total 60.09 last 48.91) - open/close 3319 (time 1782614942.73 total 70.10 last 45.55) - open/close 3826 (time 1782614952.74 total 80.12 last 50.63) - open/close 4253 (time 1782614962.74 total 90.12 last 42.69) total: 4500 open/close in 94.62 seconds: 47.56 ops/second - /unlink 1000 (time 1782614982.34 total 10.00 last 99.99) - /unlink 2035 (time 1782614992.35 total 20.01 last 103.43) - /unlink 2937 (time 1782615002.35 total 30.01 last 90.17) - /unlink 4004 (time 1782615012.35 total 40.01 last 106.68) total: 4500 /unlink in 44.29 seconds: 101.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 547 (time 1782615031.94 total 10.01 last 54.62) - open/close 1117 (time 1782615041.95 total 20.02 last 56.95) - open/close 1738 (time 1782615051.96 total 30.03 last 62.03) - open/close 2303 (time 1782615061.97 total 40.04 last 56.47) - open/close 2943 (time 1782615071.98 total 50.05 last 63.93) - open/close 3574 (time 1782615081.99 total 60.06 last 63.01) - open/close 4187 (time 1782615092.00 total 70.07 last 61.23) total: 4500 open/close in 75.59 seconds: 59.53 ops/second - /unlink 1078 (time 1782615112.30 total 10.01 last 107.73) - /unlink 2133 (time 1782615122.32 total 20.03 last 105.25) - /unlink 3262 (time 1782615132.33 total 30.03 last 112.89) - /unlink 4260 (time 1782615142.35 total 40.05 last 99.58) total: 4500 /unlink in 43.02 seconds: 104.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 535 (time 1782615161.87 total 10.01 last 53.47) - open/close 1116 (time 1782615171.89 total 20.02 last 58.00) - open/close 1735 (time 1782615181.90 total 30.03 last 61.84) - open/close 2326 (time 1782615191.91 total 40.05 last 59.01) - open/close 2922 (time 1782615201.92 total 50.05 last 59.56) - open/close 3612 (time 1782615211.92 total 60.06 last 68.98) - open/close 4207 (time 1782615221.93 total 70.06 last 59.47) total: 4500 open/close in 74.79 seconds: 60.17 ops/second - /unlink 1129 (time 1782615241.72 total 10.00 last 112.87) - /unlink 2202 (time 1782615251.73 total 20.01 last 107.22) - /unlink 3099 (time 1782615261.74 total 30.02 last 89.66) - /unlink 3993 (time 1782615271.74 total 40.02 last 89.34) total: 4500 /unlink in 44.32 seconds: 101.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 580 (time 1782615291.62 total 10.01 last 57.97) - open/close 1143 (time 1782615301.64 total 20.02 last 56.19) - open/close 1768 (time 1782615311.65 total 30.03 last 62.44) - open/close 2419 (time 1782615321.66 total 40.04 last 65.03) - open/close 2976 (time 1782615331.66 total 50.05 last 55.68) - open/close 3575 (time 1782615341.67 total 60.06 last 59.85) - open/close 4207 (time 1782615351.67 total 70.06 last 63.15) total: 4500 open/close in 74.84 seconds: 60.13 ops/second - /unlink 1103 (time 1782615372.29 total 10.00 last 110.26) - /unlink 2121 (time 1782615382.29 total 20.01 last 101.77) - /unlink 3058 (time 1782615392.30 total 30.01 last 93.63) - /unlink 4180 (time 1782615402.32 total 40.04 last 111.94) total: 4500 /unlink in 42.72 seconds: 105.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 504 (time 1782615419.73 total 10.02 last 50.31) - open/close 975 (time 1782615429.73 total 20.02 last 47.09) - open/close 1552 (time 1782615439.73 total 30.03 last 57.67) - open/close 2151 (time 1782615449.75 total 40.04 last 59.80) - open/close 2822 (time 1782615459.76 total 50.05 last 67.06) - open/close 3429 (time 1782615469.77 total 60.06 last 60.64) - open/close 4039 (time 1782615479.77 total 70.07 last 60.94) total: 4500 open/close in 78.78 seconds: 57.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1014 (time 1782615504.57 total 10.00 last 101.37) - /unlink 1973 (time 1782615514.57 total 20.01 last 95.88) - /unlink 3050 (time 1782615524.57 total 30.01 last 107.68) - /unlink 4171 (time 1782615534.59 total 40.03 last 111.88) total: 4500 /unlink in 43.07 seconds: 104.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 557 (time 1782615553.46 total 10.00 last 55.69) - open/close 1111 (time 1782615563.47 total 20.02 last 55.31) - open/close 1674 (time 1782615573.49 total 30.03 last 56.21) - open/close 2188 (time 1782615583.49 total 40.04 last 51.38) - open/close 2753 (time 1782615593.49 total 50.04 last 56.49) - open/close 3374 (time 1782615603.51 total 60.05 last 62.02) - open/close 3956 (time 1782615613.53 total 70.07 last 58.08) total: 4500 open/close in 79.70 seconds: 56.46 ops/second - /unlink 1135 (time 1782615638.49 total 10.00 last 113.45) - /unlink 2261 (time 1782615648.50 total 20.01 last 112.48) - /unlink 3338 (time 1782615658.50 total 30.02 last 107.63) - /unlink 4487 (time 1782615668.51 total 40.02 last 114.86) total: 4500 /unlink in 40.13 seconds: 112.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 595 (time 1782615684.26 total 10.01 last 59.44) - open/close 1138 (time 1782615694.29 total 20.04 last 54.14) - open/close 1770 (time 1782615704.29 total 30.04 last 63.17) - open/close 2430 (time 1782615714.30 total 40.05 last 65.94) - open/close 3071 (time 1782615724.31 total 50.06 last 64.03) - open/close 3644 (time 1782615734.32 total 60.07 last 57.24) - open/close 4305 (time 1782615744.33 total 70.08 last 66.07) total: 4500 open/close in 72.93 seconds: 61.70 ops/second - /unlink 1046 (time 1782615762.15 total 10.00 last 104.56) - /unlink 2188 (time 1782615772.15 total 20.01 last 114.12) - /unlink 3204 (time 1782615782.16 total 30.01 last 101.56) - /unlink 4398 (time 1782615792.16 total 40.01 last 119.39) total: 4500 /unlink in 40.88 seconds: 110.09 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95141 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded.