== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:35:06 (1781242506) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg323-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg323-server oleg323-server: oleg323-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg323-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg323-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 680 (time 1781242725.63 total 10.02 last 67.87) - open/close 1312 (time 1781242735.64 total 20.02 last 63.19) - open/close 2026 (time 1781242745.64 total 30.03 last 71.34) - open/close 2692 (time 1781242755.65 total 40.04 last 66.54) - open/close 3394 (time 1781242765.66 total 50.04 last 70.18) - open/close 4085 (time 1781242775.67 total 60.06 last 69.00) total: 4500 open/close in 65.93 seconds: 68.25 ops/second - /unlink 1198 (time 1781242796.18 total 10.00 last 119.78) - /unlink 2435 (time 1781242806.18 total 20.00 last 123.66) - /unlink 3708 (time 1781242816.19 total 30.01 last 127.23) total: 4500 /unlink in 36.74 seconds: 122.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 661 (time 1781242837.26 total 10.00 last 66.10) - open/close 1315 (time 1781242847.28 total 20.02 last 65.27) - open/close 1984 (time 1781242857.28 total 30.02 last 66.89) - open/close 2643 (time 1781242867.29 total 40.04 last 65.80) - open/close 3312 (time 1781242877.30 total 50.04 last 66.86) - open/close 3983 (time 1781242887.30 total 60.05 last 67.07) total: 4500 open/close in 66.99 seconds: 67.17 ops/second - /unlink 1265 (time 1781242908.74 total 10.01 last 126.43) - /unlink 2604 (time 1781242918.74 total 20.01 last 133.88) - /unlink 3935 (time 1781242928.74 total 30.01 last 133.08) total: 4500 /unlink in 34.17 seconds: 131.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 765 (time 1781242947.53 total 10.01 last 76.42) - open/close 1490 (time 1781242957.53 total 20.02 last 72.45) - open/close 2205 (time 1781242967.54 total 30.03 last 71.43) - open/close 2899 (time 1781242977.55 total 40.03 last 69.35) - open/close 3638 (time 1781242987.57 total 50.05 last 73.79) - open/close 4349 (time 1781242997.57 total 60.05 last 71.08) total: 4500 open/close in 62.09 seconds: 72.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1262 (time 1781243013.39 total 10.00 last 126.19) - /unlink 2458 (time 1781243023.40 total 20.01 last 119.55) - /unlink 3760 (time 1781243033.40 total 30.01 last 130.18) total: 4500 /unlink in 35.80 seconds: 125.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 724 (time 1781243053.09 total 10.01 last 72.35) - open/close 1464 (time 1781243063.10 total 20.02 last 73.91) - open/close 2236 (time 1781243073.10 total 30.02 last 77.20) - open/close 2959 (time 1781243083.10 total 40.02 last 72.29) - open/close 3775 (time 1781243093.11 total 50.02 last 81.57) total: 4500 open/close in 59.86 seconds: 75.18 ops/second - /unlink 1435 (time 1781243116.74 total 10.01 last 143.40) - /unlink 2876 (time 1781243126.74 total 20.01 last 144.08) - /unlink 4299 (time 1781243136.74 total 30.01 last 142.25) total: 4500 /unlink in 31.38 seconds: 143.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 772 (time 1781243151.93 total 10.00 last 77.19) - open/close 1530 (time 1781243161.94 total 20.02 last 75.67) - open/close 2247 (time 1781243171.97 total 30.05 last 71.50) - open/close 3031 (time 1781243181.98 total 40.06 last 78.30) - open/close 3848 (time 1781243191.99 total 50.07 last 81.63) total: 4500 open/close in 58.80 seconds: 76.52 ops/second - /unlink 1340 (time 1781243215.02 total 10.00 last 134.00) - /unlink 2579 (time 1781243225.02 total 20.00 last 123.86) - /unlink 3831 (time 1781243235.02 total 30.01 last 125.17) total: 4500 /unlink in 35.18 seconds: 127.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 739 (time 1781243254.05 total 10.02 last 73.78) - open/close 1521 (time 1781243264.07 total 20.04 last 78.03) - open/close 2278 (time 1781243274.08 total 30.05 last 75.63) - open/close 3025 (time 1781243284.08 total 40.05 last 74.65) - open/close 3772 (time 1781243294.09 total 50.06 last 74.67) total: 4500 open/close in 59.69 seconds: 75.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1302 (time 1781243317.04 total 10.00 last 130.19) - /unlink 2733 (time 1781243327.04 total 20.01 last 142.99) - /unlink 4208 (time 1781243337.06 total 30.02 last 147.34) total: 4500 /unlink in 31.88 seconds: 141.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 720 (time 1781243352.19 total 10.00 last 71.97) - open/close 1420 (time 1781243362.20 total 20.01 last 69.96) - open/close 2139 (time 1781243372.21 total 30.02 last 71.80) - open/close 2824 (time 1781243382.21 total 40.03 last 68.48) - open/close 3618 (time 1781243392.22 total 50.03 last 79.36) - open/close 4359 (time 1781243402.23 total 60.04 last 74.04) total: 4500 open/close in 61.79 seconds: 72.82 ops/second - /unlink 1633 (time 1781243417.26 total 10.01 last 163.18) - /unlink 3089 (time 1781243427.27 total 20.02 last 145.44) total: 4500 /unlink in 29.44 seconds: 152.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 836 (time 1781243450.70 total 10.01 last 83.48) - open/close 1707 (time 1781243460.70 total 20.02 last 87.08) - open/close 2592 (time 1781243470.72 total 30.03 last 88.35) - open/close 3411 (time 1781243480.73 total 40.04 last 81.81) - open/close 4310 (time 1781243490.74 total 50.05 last 89.86) total: 4500 open/close in 52.13 seconds: 86.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1645 (time 1781243505.82 total 10.00 last 164.49) - /unlink 3283 (time 1781243515.83 total 20.01 last 163.71) total: 4500 /unlink in 26.91 seconds: 167.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 944 (time 1781243535.60 total 10.01 last 94.31) - open/close 1813 (time 1781243545.60 total 20.01 last 86.89) - open/close 2725 (time 1781243555.61 total 30.01 last 91.16) - open/close 3608 (time 1781243565.61 total 40.02 last 88.25) - open/close 4438 (time 1781243575.62 total 50.03 last 82.89) total: 4500 open/close in 50.76 seconds: 88.66 ops/second - /unlink 1698 (time 1781243589.56 total 10.00 last 169.73) - /unlink 3309 (time 1781243599.56 total 20.00 last 161.10) total: 4500 /unlink in 27.15 seconds: 165.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 947 (time 1781243619.30 total 10.01 last 94.64) - open/close 1863 (time 1781243629.31 total 20.01 last 91.53) - open/close 2819 (time 1781243639.31 total 30.02 last 95.59) - open/close 3807 (time 1781243649.31 total 40.02 last 98.77) total: 4500 open/close in 46.88 seconds: 95.99 ops/second - /unlink 1701 (time 1781243668.94 total 10.00 last 170.08) - /unlink 3476 (time 1781243678.95 total 20.01 last 177.43) total: 4500 /unlink in 25.75 seconds: 174.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 917 (time 1781243697.68 total 10.01 last 91.64) - open/close 1921 (time 1781243707.68 total 20.01 last 100.39) - open/close 2650 (time 1781243717.69 total 30.02 last 72.82) - open/close 3218 (time 1781243727.71 total 40.04 last 56.70) - open/close 3664 (time 1781243737.72 total 50.05 last 44.56) - open/close 4157 (time 1781243747.75 total 60.08 last 49.14) total: 4500 open/close in 67.75 seconds: 66.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1224 (time 1781243772.44 total 10.00 last 122.35) - /unlink 2419 (time 1781243782.44 total 20.01 last 119.48) - /unlink 3594 (time 1781243792.44 total 30.01 last 117.50) total: 4500 /unlink in 38.61 seconds: 116.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 659 (time 1781243816.85 total 10.02 last 65.79) - open/close 1338 (time 1781243826.86 total 20.02 last 67.86) - open/close 1977 (time 1781243836.86 total 30.03 last 63.86) - open/close 2664 (time 1781243846.87 total 40.04 last 68.65) - open/close 3326 (time 1781243856.87 total 50.04 last 66.18) - open/close 3984 (time 1781243866.87 total 60.04 last 65.79) total: 4500 open/close in 67.17 seconds: 67.00 ops/second - /unlink 1315 (time 1781243887.97 total 10.00 last 131.47) - /unlink 2636 (time 1781243897.97 total 20.00 last 132.06) - /unlink 3916 (time 1781243907.97 total 30.01 last 128.00) total: 4500 /unlink in 34.68 seconds: 129.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 696 (time 1781243927.31 total 10.01 last 69.52) - open/close 1345 (time 1781243937.32 total 20.02 last 64.86) - open/close 2023 (time 1781243947.33 total 30.03 last 67.74) - open/close 2706 (time 1781243957.34 total 40.04 last 68.21) - open/close 3400 (time 1781243967.35 total 50.05 last 69.37) - open/close 4062 (time 1781243977.35 total 60.05 last 66.20) total: 4500 open/close in 66.39 seconds: 67.78 ops/second - /unlink 1240 (time 1781243998.33 total 10.00 last 123.95) - /unlink 2332 (time 1781244008.34 total 20.01 last 109.16) - /unlink 3594 (time 1781244018.35 total 30.02 last 126.06) total: 4500 /unlink in 38.57 seconds: 116.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 575 (time 1781244042.41 total 10.02 last 57.41) - open/close 1261 (time 1781244052.43 total 20.03 last 68.51) - open/close 1941 (time 1781244062.44 total 30.04 last 67.90) - open/close 2583 (time 1781244072.45 total 40.05 last 64.16) - open/close 3244 (time 1781244082.46 total 50.06 last 66.04) - open/close 3857 (time 1781244092.49 total 60.09 last 61.11) total: 4500 open/close in 69.30 seconds: 64.93 ops/second - /unlink 1249 (time 1781244116.75 total 10.00 last 124.84) - /unlink 2426 (time 1781244126.77 total 20.02 last 117.52) - /unlink 3582 (time 1781244136.77 total 30.02 last 115.56) total: 4500 /unlink in 37.54 seconds: 119.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 549 (time 1781244159.19 total 10.01 last 54.86) - open/close 1158 (time 1781244169.19 total 20.01 last 60.87) - open/close 1772 (time 1781244179.20 total 30.02 last 61.34) - open/close 2412 (time 1781244189.21 total 40.03 last 63.98) - open/close 3097 (time 1781244199.22 total 50.04 last 68.44) - open/close 3744 (time 1781244209.23 total 60.04 last 64.64) - open/close 4408 (time 1781244219.24 total 70.06 last 66.28) total: 4500 open/close in 71.33 seconds: 63.09 ops/second - /unlink 1164 (time 1781244235.07 total 10.00 last 116.39) - /unlink 2334 (time 1781244245.08 total 20.01 last 116.91) - /unlink 3593 (time 1781244255.08 total 30.01 last 125.86) total: 4500 /unlink in 37.61 seconds: 119.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 727 (time 1781244278.66 total 10.02 last 72.56) - open/close 1350 (time 1781244288.67 total 20.03 last 62.27) - open/close 1982 (time 1781244298.67 total 30.03 last 63.19) - open/close 2657 (time 1781244308.68 total 40.04 last 67.43) - open/close 3262 (time 1781244318.70 total 50.06 last 60.34) - open/close 3987 (time 1781244328.72 total 60.08 last 72.41) total: 4500 open/close in 68.31 seconds: 65.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1264 (time 1781244351.85 total 10.01 last 126.33) - /unlink 2447 (time 1781244361.85 total 20.01 last 118.27) - /unlink 3521 (time 1781244371.86 total 30.01 last 107.37) total: 4500 /unlink in 38.55 seconds: 116.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 604 (time 1781244395.92 total 10.01 last 60.36) - open/close 1188 (time 1781244405.94 total 20.02 last 58.31) - open/close 1756 (time 1781244415.95 total 30.03 last 56.73) - open/close 2416 (time 1781244425.96 total 40.04 last 65.94) - open/close 3117 (time 1781244435.97 total 50.05 last 70.03) - open/close 3739 (time 1781244445.97 total 60.05 last 62.19) - open/close 4437 (time 1781244455.97 total 70.06 last 69.79) total: 4500 open/close in 71.02 seconds: 63.36 ops/second - /unlink 1201 (time 1781244471.71 total 10.00 last 120.04) - /unlink 2358 (time 1781244481.72 total 20.01 last 115.66) - /unlink 3519 (time 1781244491.72 total 30.01 last 116.07) total: 4500 /unlink in 38.53 seconds: 116.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 699 (time 1781244515.98 total 10.00 last 69.89) - open/close 1439 (time 1781244525.99 total 20.00 last 73.99) - open/close 2075 (time 1781244535.99 total 30.00 last 63.59) - open/close 2737 (time 1781244546.00 total 40.02 last 66.11) - open/close 3386 (time 1781244556.01 total 50.02 last 64.86) - open/close 4021 (time 1781244566.01 total 60.03 last 63.49) total: 4500 open/close in 69.44 seconds: 64.80 ops/second - /unlink 844 (time 1781244592.51 total 10.00 last 84.36) - /unlink 1805 (time 1781244602.52 total 20.02 last 95.99) - /unlink 2995 (time 1781244612.53 total 30.02 last 118.97) - /unlink 4187 (time 1781244622.53 total 40.02 last 119.18) total: 4500 /unlink in 42.85 seconds: 105.03 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94031 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: modules unloaded.