== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:26:06 (1787138766) Stopping clients: oleg158-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg158-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg158-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg158-server oleg158-server: oleg158-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg158-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 703 (time 1787138985.45 total 10.00 last 70.30) - open/close 1329 (time 1787138995.46 total 20.01 last 62.53) - open/close 1867 (time 1787139005.46 total 30.01 last 53.80) - open/close 2418 (time 1787139015.46 total 40.01 last 55.09) - open/close 3025 (time 1787139025.50 total 50.05 last 60.49) - open/close 3666 (time 1787139035.50 total 60.05 last 64.10) - open/close 4243 (time 1787139045.50 total 70.05 last 57.69) total: 4500 open/close in 73.97 seconds: 60.83 ops/second - /unlink 998 (time 1787139065.22 total 10.00 last 99.76) - /unlink 2138 (time 1787139075.23 total 20.01 last 113.93) - /unlink 3295 (time 1787139085.23 total 30.01 last 115.66) - /unlink 4446 (time 1787139095.25 total 40.03 last 114.89) total: 4500 /unlink in 40.71 seconds: 110.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 563 (time 1787139112.36 total 10.02 last 56.17) - open/close 1162 (time 1787139122.37 total 20.03 last 59.87) - open/close 1805 (time 1787139132.37 total 30.03 last 64.28) - open/close 2469 (time 1787139142.38 total 40.04 last 66.32) - open/close 3113 (time 1787139152.40 total 50.06 last 64.27) - open/close 3828 (time 1787139162.40 total 60.07 last 71.48) - open/close 4474 (time 1787139172.41 total 70.07 last 64.56) total: 4500 open/close in 70.56 seconds: 63.78 ops/second - /unlink 1153 (time 1787139188.42 total 10.03 last 114.93) - /unlink 2114 (time 1787139198.42 total 20.04 last 96.02) - /unlink 3094 (time 1787139208.43 total 30.05 last 97.94) - /unlink 4231 (time 1787139218.44 total 40.06 last 113.57) total: 4500 /unlink in 42.31 seconds: 106.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 637 (time 1787139236.28 total 10.02 last 63.56) - open/close 1278 (time 1787139246.30 total 20.04 last 63.99) - open/close 1971 (time 1787139256.31 total 30.05 last 69.22) - open/close 2572 (time 1787139266.33 total 40.07 last 59.97) - open/close 3138 (time 1787139276.33 total 50.07 last 56.59) - open/close 3725 (time 1787139286.35 total 60.09 last 58.63) - open/close 4322 (time 1787139296.38 total 70.12 last 59.51) total: 4500 open/close in 72.45 seconds: 62.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1198 (time 1787139313.74 total 10.00 last 119.77) - /unlink 2486 (time 1787139323.74 total 20.01 last 128.73) - /unlink 3753 (time 1787139333.75 total 30.01 last 126.64) total: 4500 /unlink in 36.03 seconds: 124.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 745 (time 1787139355.18 total 10.00 last 74.48) - open/close 1552 (time 1787139365.19 total 20.01 last 80.65) - open/close 2325 (time 1787139375.19 total 30.01 last 77.29) - open/close 3104 (time 1787139385.19 total 40.02 last 77.84) - open/close 3895 (time 1787139395.21 total 50.03 last 79.01) total: 4500 open/close in 58.99 seconds: 76.28 ops/second - /unlink 1313 (time 1787139419.95 total 10.00 last 131.30) - /unlink 2568 (time 1787139429.96 total 20.00 last 125.44) - /unlink 3869 (time 1787139439.96 total 30.01 last 130.02) total: 4500 /unlink in 34.97 seconds: 128.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 693 (time 1787139460.40 total 10.01 last 69.20) - open/close 1385 (time 1787139470.41 total 20.02 last 69.16) - open/close 2125 (time 1787139480.43 total 30.04 last 73.84) - open/close 2774 (time 1787139490.44 total 40.05 last 64.85) - open/close 3488 (time 1787139500.46 total 50.08 last 71.21) - open/close 4248 (time 1787139510.47 total 60.08 last 75.96) total: 4500 open/close in 63.58 seconds: 70.78 ops/second - /unlink 1554 (time 1787139529.62 total 10.00 last 155.35) - /unlink 2876 (time 1787139539.62 total 20.01 last 132.16) - /unlink 4183 (time 1787139549.62 total 30.01 last 130.69) total: 4500 /unlink in 32.42 seconds: 138.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 702 (time 1787139567.27 total 10.00 last 70.20) - open/close 1428 (time 1787139577.27 total 20.00 last 72.59) - open/close 2164 (time 1787139587.29 total 30.02 last 73.46) - open/close 2901 (time 1787139597.29 total 40.03 last 73.66) - open/close 3646 (time 1787139607.30 total 50.03 last 74.43) - open/close 4345 (time 1787139617.32 total 60.05 last 69.81) total: 4500 open/close in 62.02 seconds: 72.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1123 (time 1787139635.60 total 10.00 last 112.27) - /unlink 2262 (time 1787139645.61 total 20.01 last 113.84) - /unlink 3496 (time 1787139655.61 total 30.01 last 123.36) total: 4500 /unlink in 38.65 seconds: 116.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 683 (time 1787139680.24 total 10.03 last 68.07) - open/close 1364 (time 1787139690.24 total 20.04 last 68.06) - open/close 2030 (time 1787139700.25 total 30.05 last 66.55) - open/close 2716 (time 1787139710.26 total 40.05 last 68.57) - open/close 3451 (time 1787139720.26 total 50.05 last 73.50) - open/close 4182 (time 1787139730.27 total 60.07 last 72.97) total: 4500 open/close in 65.47 seconds: 68.74 ops/second - /unlink 1351 (time 1787139751.83 total 10.00 last 135.10) - /unlink 2610 (time 1787139761.83 total 20.00 last 125.87) - /unlink 4103 (time 1787139771.84 total 30.00 last 149.26) total: 4500 /unlink in 32.78 seconds: 137.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 666 (time 1787139789.67 total 10.00 last 66.59) - open/close 1482 (time 1787139799.70 total 20.02 last 81.41) - open/close 2274 (time 1787139809.70 total 30.03 last 79.13) - open/close 2887 (time 1787139819.71 total 40.04 last 61.24) - open/close 3583 (time 1787139829.72 total 50.05 last 69.54) - open/close 4275 (time 1787139839.73 total 60.06 last 69.14) total: 4500 open/close in 62.75 seconds: 71.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1260 (time 1787139857.43 total 10.00 last 125.94) - /unlink 2648 (time 1787139867.44 total 20.01 last 138.70) - /unlink 4051 (time 1787139877.44 total 30.01 last 140.29) total: 4500 /unlink in 33.60 seconds: 133.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 709 (time 1787139897.23 total 10.01 last 70.86) - open/close 1405 (time 1787139907.24 total 20.02 last 69.51) - open/close 2015 (time 1787139917.25 total 30.03 last 60.91) - open/close 2749 (time 1787139927.30 total 40.08 last 73.05) - open/close 3406 (time 1787139937.30 total 50.08 last 65.70) - open/close 4026 (time 1787139947.31 total 60.09 last 61.97) total: 4500 open/close in 66.84 seconds: 67.32 ops/second - /unlink 1258 (time 1787139969.00 total 10.00 last 125.80) - /unlink 2571 (time 1787139979.00 total 20.01 last 131.22) - /unlink 3939 (time 1787139989.01 total 30.01 last 136.76) total: 4500 /unlink in 34.24 seconds: 131.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 815 (time 1787140008.12 total 10.02 last 81.36) - open/close 1613 (time 1787140018.12 total 20.02 last 79.79) - open/close 2387 (time 1787140028.14 total 30.03 last 77.30) - open/close 3123 (time 1787140038.14 total 40.04 last 73.54) - open/close 3860 (time 1787140048.15 total 50.05 last 73.63) total: 4500 open/close in 59.32 seconds: 75.86 ops/second - /unlink 1428 (time 1787140072.07 total 10.00 last 142.79) - /unlink 2910 (time 1787140082.09 total 20.02 last 147.94) - /unlink 4398 (time 1787140092.09 total 30.02 last 148.79) total: 4500 /unlink in 30.88 seconds: 145.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 710 (time 1787140107.58 total 10.01 last 70.95) - open/close 1449 (time 1787140117.58 total 20.01 last 73.89) - open/close 2048 (time 1787140127.58 total 30.01 last 59.88) - open/close 2722 (time 1787140137.60 total 40.03 last 67.30) - open/close 3274 (time 1787140147.60 total 50.03 last 55.19) - open/close 3899 (time 1787140157.60 total 60.04 last 62.46) - open/close 4347 (time 1787140167.61 total 70.04 last 44.77) total: 4500 open/close in 72.83 seconds: 61.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1250 (time 1787140185.89 total 10.00 last 124.97) - /unlink 2444 (time 1787140195.89 total 20.00 last 119.37) - /unlink 3562 (time 1787140205.90 total 30.01 last 111.74) - /unlink 4453 (time 1787140215.91 total 40.02 last 89.04) total: 4500 /unlink in 40.50 seconds: 111.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 476 (time 1787140233.93 total 10.02 last 47.50) - open/close 1045 (time 1787140243.93 total 20.03 last 56.87) - open/close 1685 (time 1787140253.93 total 30.03 last 63.99) - open/close 2307 (time 1787140263.94 total 40.03 last 62.17) - open/close 2877 (time 1787140273.96 total 50.05 last 56.90) - open/close 3577 (time 1787140283.96 total 60.06 last 69.95) - open/close 4276 (time 1787140293.97 total 70.06 last 69.88) total: 4500 open/close in 73.58 seconds: 61.16 ops/second - /unlink 1184 (time 1787140313.74 total 10.00 last 118.37) - /unlink 2242 (time 1787140323.75 total 20.01 last 105.75) - /unlink 3370 (time 1787140333.75 total 30.01 last 112.78) - /unlink 4379 (time 1787140343.75 total 40.01 last 100.88) total: 4500 /unlink in 41.58 seconds: 108.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 590 (time 1787140362.04 total 10.00 last 58.99) - open/close 1282 (time 1787140372.05 total 20.01 last 69.12) - open/close 1863 (time 1787140382.06 total 30.02 last 58.04) - open/close 2636 (time 1787140392.07 total 40.03 last 77.22) - open/close 3342 (time 1787140402.09 total 50.05 last 70.46) - open/close 3978 (time 1787140412.09 total 60.06 last 63.57) total: 4500 open/close in 67.37 seconds: 66.80 ops/second - /unlink 1285 (time 1787140433.99 total 10.01 last 128.43) - /unlink 2404 (time 1787140444.02 total 20.03 last 111.65) - /unlink 3589 (time 1787140454.02 total 30.03 last 118.48) - /unlink 4447 (time 1787140464.02 total 40.03 last 85.77) total: 4500 /unlink in 40.99 seconds: 109.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 734 (time 1787140481.92 total 10.00 last 73.36) - open/close 1248 (time 1787140491.93 total 20.01 last 51.35) - open/close 1890 (time 1787140501.93 total 30.01 last 64.20) - open/close 2534 (time 1787140511.94 total 40.02 last 64.37) - open/close 3104 (time 1787140521.95 total 50.03 last 56.94) - open/close 3643 (time 1787140531.95 total 60.03 last 53.87) - open/close 4205 (time 1787140541.96 total 70.04 last 56.16) total: 4500 open/close in 74.47 seconds: 60.43 ops/second - /unlink 1271 (time 1787140562.03 total 10.01 last 126.96) - /unlink 2299 (time 1787140572.03 total 20.02 last 102.73) - /unlink 3299 (time 1787140582.04 total 30.03 last 99.92) - /unlink 4130 (time 1787140592.05 total 40.03 last 83.04) total: 4500 /unlink in 43.63 seconds: 103.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 523 (time 1787140613.77 total 10.02 last 52.22) - open/close 1104 (time 1787140623.78 total 20.02 last 58.08) - open/close 1694 (time 1787140633.78 total 30.02 last 58.99) - open/close 2297 (time 1787140643.80 total 40.04 last 60.18) - open/close 2775 (time 1787140653.82 total 50.06 last 47.72) - open/close 3356 (time 1787140663.83 total 60.07 last 58.03) - open/close 3956 (time 1787140673.87 total 70.11 last 59.76) total: 4500 open/close in 77.34 seconds: 58.18 ops/second - /unlink 1143 (time 1787140697.43 total 10.01 last 114.20) - /unlink 2016 (time 1787140707.43 total 20.01 last 87.29) - /unlink 3043 (time 1787140717.44 total 30.02 last 102.63) - /unlink 4116 (time 1787140727.44 total 40.02 last 107.30) total: 4500 /unlink in 43.72 seconds: 102.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 534 (time 1787140748.15 total 10.01 last 53.37) - open/close 1089 (time 1787140758.15 total 20.01 last 55.48) - open/close 1788 (time 1787140768.16 total 30.02 last 69.84) - open/close 2296 (time 1787140778.20 total 40.06 last 50.61) - open/close 2908 (time 1787140788.21 total 50.07 last 61.13) - open/close 3554 (time 1787140798.22 total 60.08 last 64.56) - open/close 4263 (time 1787140808.22 total 70.08 last 70.89) total: 4500 open/close in 74.06 seconds: 60.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1383 (time 1787140827.13 total 10.00 last 138.29) - /unlink 2563 (time 1787140837.14 total 20.01 last 117.93) - /unlink 3620 (time 1787140847.14 total 30.01 last 105.65) total: 4500 /unlink in 37.60 seconds: 119.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 629 (time 1787140870.55 total 10.01 last 62.87) - open/close 1258 (time 1787140880.57 total 20.02 last 62.80) - open/close 1842 (time 1787140890.59 total 30.04 last 58.31) - open/close 2524 (time 1787140900.60 total 40.05 last 68.11) - open/close 3028 (time 1787140910.61 total 50.06 last 50.34) - open/close 3589 (time 1787140920.61 total 60.07 last 56.08) - open/close 4197 (time 1787140930.63 total 70.08 last 60.73) total: 4500 open/close in 74.77 seconds: 60.19 ops/second - /unlink 1132 (time 1787140951.15 total 10.00 last 113.18) - /unlink 2330 (time 1787140961.16 total 20.02 last 119.64) - /unlink 3431 (time 1787140971.19 total 30.04 last 109.79) - /unlink 4475 (time 1787140981.19 total 40.05 last 104.36) total: 4500 /unlink in 40.37 seconds: 111.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 702 (time 1787140997.13 total 10.00 last 70.17) - open/close 1399 (time 1787141007.13 total 20.01 last 69.68) - open/close 2094 (time 1787141017.13 total 30.01 last 69.47) - open/close 2866 (time 1787141027.14 total 40.02 last 77.17) - open/close 3557 (time 1787141037.16 total 50.04 last 68.96) - open/close 4305 (time 1787141047.16 total 60.04 last 74.79) total: 4500 open/close in 62.28 seconds: 72.26 ops/second - /unlink 1506 (time 1787141063.77 total 10.00 last 150.56) - /unlink 2784 (time 1787141073.78 total 20.01 last 127.72) - /unlink 4021 (time 1787141083.78 total 30.01 last 123.68) total: 4500 /unlink in 34.43 seconds: 130.71 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95279 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: modules unloaded.