== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:23:55 (1787559835) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg204-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg204-server oleg204-server: oleg204-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 493 (time 1787560048.97 total 10.04 last 49.11) - open/close 1185 (time 1787560058.98 total 20.05 last 69.16) - open/close 1883 (time 1787560068.98 total 30.05 last 69.79) - open/close 2584 (time 1787560078.99 total 40.06 last 70.03) - open/close 3281 (time 1787560088.99 total 50.06 last 69.69) - open/close 3995 (time 1787560098.99 total 60.06 last 71.39) total: 4500 open/close in 67.89 seconds: 66.29 ops/second - /unlink 1308 (time 1787560124.33 total 10.01 last 130.62) - /unlink 2825 (time 1787560134.33 total 20.02 last 151.67) - /unlink 4276 (time 1787560144.34 total 30.02 last 145.06) total: 4500 /unlink in 31.76 seconds: 141.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 653 (time 1787560160.93 total 10.00 last 65.28) - open/close 1345 (time 1787560170.95 total 20.02 last 69.07) - open/close 2100 (time 1787560180.95 total 30.03 last 75.47) - open/close 2771 (time 1787560190.96 total 40.03 last 67.05) - open/close 3551 (time 1787560200.96 total 50.04 last 77.97) - open/close 4426 (time 1787560210.98 total 60.06 last 87.32) total: 4500 open/close in 61.14 seconds: 73.60 ops/second - /unlink 1532 (time 1787560226.84 total 10.01 last 153.11) - /unlink 2551 (time 1787560236.86 total 20.02 last 101.76) - /unlink 3430 (time 1787560246.86 total 30.02 last 87.90) total: 4500 /unlink in 38.46 seconds: 116.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 611 (time 1787560270.97 total 10.02 last 61.00) - open/close 1045 (time 1787560280.97 total 20.02 last 43.38) - open/close 1622 (time 1787560291.00 total 30.05 last 57.52) - open/close 2234 (time 1787560301.02 total 40.06 last 61.11) - open/close 2844 (time 1787560311.03 total 50.08 last 60.91) - open/close 3613 (time 1787560321.05 total 60.10 last 76.74) - open/close 4228 (time 1787560331.07 total 70.12 last 61.40) total: 4500 open/close in 74.97 seconds: 60.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1107 (time 1787560351.94 total 10.02 last 110.49) - /unlink 2187 (time 1787560361.94 total 20.03 last 107.91) - /unlink 3144 (time 1787560371.95 total 30.03 last 95.64) - /unlink 4444 (time 1787560381.95 total 40.04 last 129.96) total: 4500 /unlink in 40.71 seconds: 110.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 720 (time 1787560397.49 total 10.01 last 71.95) - open/close 1492 (time 1787560407.50 total 20.02 last 77.13) - open/close 2266 (time 1787560417.51 total 30.03 last 77.32) - open/close 3022 (time 1787560427.51 total 40.03 last 75.59) - open/close 3759 (time 1787560437.52 total 50.03 last 73.65) total: 4500 open/close in 59.83 seconds: 75.21 ops/second - /unlink 1644 (time 1787560462.10 total 10.00 last 164.35) - /unlink 3097 (time 1787560472.12 total 20.02 last 145.11) total: 4500 /unlink in 28.71 seconds: 156.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 827 (time 1787560495.69 total 10.01 last 82.64) - open/close 1592 (time 1787560505.71 total 20.02 last 76.39) - open/close 2193 (time 1787560515.74 total 30.05 last 59.92) - open/close 2805 (time 1787560525.76 total 40.07 last 61.08) - open/close 3563 (time 1787560535.77 total 50.08 last 75.74) - open/close 4278 (time 1787560545.78 total 60.10 last 71.39) total: 4500 open/close in 63.58 seconds: 70.78 ops/second - /unlink 1361 (time 1787560564.88 total 10.01 last 135.96) - /unlink 2471 (time 1787560574.89 total 20.02 last 110.85) - /unlink 3561 (time 1787560584.91 total 30.04 last 108.81) total: 4500 /unlink in 38.41 seconds: 117.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 630 (time 1787560609.20 total 10.03 last 62.82) - open/close 1328 (time 1787560619.22 total 20.04 last 69.72) - open/close 2019 (time 1787560629.22 total 30.05 last 69.05) - open/close 2638 (time 1787560639.23 total 40.05 last 61.88) - open/close 3257 (time 1787560649.23 total 50.05 last 61.88) - open/close 4053 (time 1787560659.23 total 60.05 last 79.59) total: 4500 open/close in 66.02 seconds: 68.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1388 (time 1787560680.95 total 10.00 last 138.75) - /unlink 2754 (time 1787560690.96 total 20.01 last 136.54) - /unlink 4106 (time 1787560700.96 total 30.01 last 135.14) total: 4500 /unlink in 32.62 seconds: 137.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 727 (time 1787560718.48 total 10.00 last 72.67) - open/close 1438 (time 1787560728.51 total 20.03 last 70.88) - open/close 2077 (time 1787560738.53 total 30.05 last 63.82) - open/close 2595 (time 1787560748.55 total 40.07 last 51.67) - open/close 3114 (time 1787560758.56 total 50.08 last 51.85) - open/close 3780 (time 1787560768.56 total 60.08 last 66.60) - open/close 4377 (time 1787560778.56 total 70.09 last 59.67) total: 4500 open/close in 71.99 seconds: 62.51 ops/second - /unlink 1632 (time 1787560795.28 total 10.02 last 162.94) - /unlink 3156 (time 1787560805.29 total 20.03 last 152.20) - /unlink 4407 (time 1787560815.33 total 30.07 last 124.61) total: 4500 /unlink in 31.06 seconds: 144.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 775 (time 1787560831.31 total 10.01 last 77.45) - open/close 1417 (time 1787560841.31 total 20.01 last 64.17) - open/close 2131 (time 1787560851.32 total 30.02 last 71.34) - open/close 2761 (time 1787560861.32 total 40.02 last 63.00) - open/close 3528 (time 1787560871.33 total 50.02 last 76.68) - open/close 4374 (time 1787560881.33 total 60.02 last 84.59) total: 4500 open/close in 61.31 seconds: 73.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1633 (time 1787560896.92 total 10.00 last 163.28) - /unlink 3188 (time 1787560906.93 total 20.00 last 155.46) total: 4500 /unlink in 29.15 seconds: 154.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 748 (time 1787560930.93 total 10.00 last 74.79) - open/close 1657 (time 1787560940.94 total 20.01 last 90.81) - open/close 2538 (time 1787560950.94 total 30.01 last 88.10) - open/close 3415 (time 1787560960.94 total 40.01 last 87.68) - open/close 4099 (time 1787560970.95 total 50.02 last 68.38) total: 4500 open/close in 54.92 seconds: 81.93 ops/second - /unlink 1326 (time 1787560990.60 total 10.00 last 132.60) - /unlink 2673 (time 1787561000.60 total 20.00 last 134.69) - /unlink 3886 (time 1787561010.61 total 30.01 last 121.25) total: 4500 /unlink in 34.24 seconds: 131.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 807 (time 1787561030.55 total 10.01 last 80.63) - open/close 1639 (time 1787561040.57 total 20.03 last 83.02) - open/close 2395 (time 1787561050.58 total 30.04 last 75.54) - open/close 3113 (time 1787561060.58 total 40.04 last 71.80) - open/close 3844 (time 1787561070.59 total 50.05 last 73.05) total: 4500 open/close in 58.22 seconds: 77.30 ops/second - /unlink 1599 (time 1787561093.65 total 10.01 last 159.78) - /unlink 2974 (time 1787561103.66 total 20.01 last 137.47) - /unlink 4452 (time 1787561113.66 total 30.01 last 147.74) total: 4500 /unlink in 30.52 seconds: 147.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 760 (time 1787561128.11 total 10.01 last 75.95) - open/close 1422 (time 1787561138.12 total 20.01 last 66.18) - open/close 2010 (time 1787561148.13 total 30.02 last 58.74) - open/close 2489 (time 1787561158.14 total 40.03 last 47.85) - open/close 3260 (time 1787561168.14 total 50.03 last 77.10) - open/close 3928 (time 1787561178.14 total 60.04 last 66.76) - open/close 4479 (time 1787561188.15 total 70.04 last 55.09) total: 4500 open/close in 70.29 seconds: 64.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1556 (time 1787561203.77 total 10.00 last 155.59) - /unlink 3117 (time 1787561213.77 total 20.00 last 156.07) total: 4500 /unlink in 28.64 seconds: 157.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 722 (time 1787561237.49 total 10.01 last 72.15) - open/close 1441 (time 1787561247.49 total 20.01 last 71.87) - open/close 2210 (time 1787561257.50 total 30.02 last 76.81) - open/close 2876 (time 1787561267.52 total 40.04 last 66.52) - open/close 3741 (time 1787561277.53 total 50.04 last 86.43) total: 4500 open/close in 59.13 seconds: 76.10 ops/second - /unlink 1363 (time 1787561301.12 total 10.00 last 136.27) - /unlink 2952 (time 1787561311.13 total 20.02 last 158.63) total: 4500 /unlink in 29.61 seconds: 151.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 570 (time 1787561336.30 total 10.01 last 56.94) - open/close 1198 (time 1787561346.31 total 20.02 last 62.74) - open/close 1845 (time 1787561356.32 total 30.03 last 64.66) - open/close 2509 (time 1787561366.32 total 40.03 last 66.36) - open/close 3116 (time 1787561376.33 total 50.05 last 60.63) - open/close 3884 (time 1787561386.34 total 60.05 last 76.75) total: 4500 open/close in 67.16 seconds: 67.00 ops/second - /unlink 1490 (time 1787561407.56 total 10.00 last 148.99) - /unlink 2851 (time 1787561417.56 total 20.01 last 136.03) - /unlink 4016 (time 1787561427.57 total 30.02 last 116.39) total: 4500 /unlink in 34.19 seconds: 131.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 682 (time 1787561450.00 total 10.01 last 68.11) - open/close 1367 (time 1787561460.00 total 20.02 last 68.47) - open/close 2143 (time 1787561470.01 total 30.02 last 77.57) - open/close 2833 (time 1787561480.01 total 40.02 last 68.97) - open/close 3460 (time 1787561490.01 total 50.03 last 62.69) - open/close 4337 (time 1787561500.02 total 60.03 last 87.68) total: 4500 open/close in 62.10 seconds: 72.46 ops/second - /unlink 1351 (time 1787561517.19 total 10.00 last 135.09) - /unlink 2840 (time 1787561527.19 total 20.00 last 148.88) - /unlink 4334 (time 1787561537.19 total 30.00 last 149.40) total: 4500 /unlink in 31.18 seconds: 144.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 821 (time 1787561554.16 total 10.01 last 82.03) - open/close 1576 (time 1787561564.17 total 20.01 last 75.49) - open/close 2292 (time 1787561574.17 total 30.02 last 71.56) - open/close 2925 (time 1787561584.17 total 40.02 last 63.29) - open/close 3750 (time 1787561594.18 total 50.02 last 82.48) total: 4500 open/close in 59.01 seconds: 76.26 ops/second - /unlink 1635 (time 1787561617.24 total 10.00 last 163.43) - /unlink 3164 (time 1787561627.24 total 20.01 last 152.87) total: 4500 /unlink in 28.44 seconds: 158.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 852 (time 1787561650.33 total 10.00 last 85.18) - open/close 1629 (time 1787561660.36 total 20.03 last 77.52) - open/close 2619 (time 1787561670.37 total 30.03 last 98.91) - open/close 3593 (time 1787561680.37 total 40.04 last 97.38) - open/close 4459 (time 1787561690.38 total 50.04 last 86.54) total: 4500 open/close in 50.64 seconds: 88.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1291 (time 1787561705.57 total 10.00 last 129.10) - /unlink 2749 (time 1787561715.57 total 20.00 last 145.75) - /unlink 4049 (time 1787561725.57 total 30.00 last 130.00) total: 4500 /unlink in 33.79 seconds: 133.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 730 (time 1787561745.78 total 10.00 last 72.98) - open/close 1537 (time 1787561755.78 total 20.01 last 80.67) - open/close 2337 (time 1787561765.79 total 30.01 last 79.96) - open/close 3124 (time 1787561775.80 total 40.02 last 78.61) - open/close 3958 (time 1787561785.83 total 50.05 last 83.14) total: 4500 open/close in 56.31 seconds: 79.91 ops/second - /unlink 1522 (time 1787561806.72 total 10.01 last 152.12) - /unlink 3149 (time 1787561816.72 total 20.01 last 162.68) total: 4500 /unlink in 29.03 seconds: 155.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 817 (time 1787561841.05 total 10.01 last 81.62) - open/close 1530 (time 1787561851.05 total 20.01 last 71.27) - open/close 2044 (time 1787561861.06 total 30.02 last 51.35) - open/close 2774 (time 1787561871.09 total 40.05 last 72.79) - open/close 3520 (time 1787561881.10 total 50.07 last 74.50) - open/close 4312 (time 1787561891.11 total 60.07 last 79.14) total: 4500 open/close in 61.97 seconds: 72.62 ops/second - /unlink 1476 (time 1787561907.18 total 10.00 last 147.60) - /unlink 2745 (time 1787561917.19 total 20.02 last 126.69) - /unlink 3778 (time 1787561927.20 total 30.02 last 103.25) total: 4500 /unlink in 34.90 seconds: 128.94 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95332 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded.