== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:14:00 (1781230440) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg223-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 743 (time 1781230637.01 total 10.00 last 74.27) - open/close 1519 (time 1781230647.02 total 20.01 last 77.54) - open/close 2288 (time 1781230657.03 total 30.02 last 76.83) - open/close 3061 (time 1781230667.04 total 40.03 last 77.25) - open/close 3822 (time 1781230677.05 total 50.04 last 76.00) total: 4500 open/close in 59.04 seconds: 76.21 ops/second - /unlink 1316 (time 1781230700.66 total 10.00 last 131.59) - /unlink 2664 (time 1781230710.67 total 20.00 last 134.78) - /unlink 4036 (time 1781230720.67 total 30.00 last 137.18) total: 4500 /unlink in 33.84 seconds: 133.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 741 (time 1781230739.39 total 10.01 last 74.05) - open/close 1465 (time 1781230749.39 total 20.01 last 72.36) - open/close 2222 (time 1781230759.40 total 30.02 last 75.66) - open/close 2755 (time 1781230769.40 total 40.02 last 53.28) - open/close 3315 (time 1781230779.41 total 50.03 last 55.97) - open/close 4032 (time 1781230789.41 total 60.03 last 71.66) total: 4500 open/close in 66.66 seconds: 67.50 ops/second - /unlink 1003 (time 1781230810.97 total 10.02 last 100.12) - /unlink 2243 (time 1781230820.98 total 20.02 last 123.97) - /unlink 3654 (time 1781230830.98 total 30.02 last 141.08) total: 4500 /unlink in 36.38 seconds: 123.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 672 (time 1781230852.33 total 10.02 last 67.06) - open/close 1313 (time 1781230862.33 total 20.02 last 64.10) - open/close 1996 (time 1781230872.35 total 30.03 last 68.22) - open/close 2727 (time 1781230882.35 total 40.04 last 73.08) - open/close 3544 (time 1781230892.36 total 50.04 last 81.63) - open/close 4336 (time 1781230902.36 total 60.04 last 79.19) total: 4500 open/close in 62.00 seconds: 72.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1381 (time 1781230918.38 total 10.00 last 138.05) - /unlink 2732 (time 1781230928.39 total 20.01 last 135.08) - /unlink 3933 (time 1781230938.41 total 30.02 last 119.87) total: 4500 /unlink in 34.31 seconds: 131.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 497 (time 1781230959.91 total 10.02 last 49.62) - open/close 1084 (time 1781230969.91 total 20.02 last 58.67) - open/close 1852 (time 1781230979.92 total 30.03 last 76.73) - open/close 2566 (time 1781230989.93 total 40.03 last 71.37) - open/close 3308 (time 1781230999.95 total 50.06 last 74.04) - open/close 4094 (time 1781231009.95 total 60.06 last 78.60) total: 4500 open/close in 66.40 seconds: 67.77 ops/second - /unlink 1149 (time 1781231032.08 total 10.01 last 114.83) - /unlink 2539 (time 1781231042.08 total 20.01 last 139.00) - /unlink 3869 (time 1781231052.09 total 30.01 last 132.92) total: 4500 /unlink in 34.00 seconds: 132.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 764 (time 1781231070.86 total 10.01 last 76.30) - open/close 1462 (time 1781231080.87 total 20.02 last 69.78) - open/close 2212 (time 1781231090.87 total 30.02 last 74.99) - open/close 2889 (time 1781231100.88 total 40.03 last 67.61) - open/close 3636 (time 1781231110.88 total 50.03 last 74.68) - open/close 4283 (time 1781231120.89 total 60.04 last 64.65) total: 4500 open/close in 63.43 seconds: 70.95 ops/second - /unlink 1235 (time 1781231138.66 total 10.00 last 123.44) - /unlink 2559 (time 1781231148.66 total 20.01 last 132.39) - /unlink 3972 (time 1781231158.67 total 30.01 last 141.18) total: 4500 /unlink in 34.19 seconds: 131.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 664 (time 1781231177.62 total 10.01 last 66.36) - open/close 1479 (time 1781231187.62 total 20.01 last 81.49) - open/close 2169 (time 1781231197.64 total 30.02 last 68.91) - open/close 2820 (time 1781231207.65 total 40.03 last 65.05) - open/close 3456 (time 1781231217.66 total 50.04 last 63.51) - open/close 4188 (time 1781231227.67 total 60.05 last 73.12) total: 4500 open/close in 64.65 seconds: 69.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 995 (time 1781231246.98 total 10.00 last 99.49) - /unlink 2168 (time 1781231256.98 total 20.00 last 117.29) - /unlink 3265 (time 1781231266.99 total 30.01 last 109.63) total: 4500 /unlink in 39.89 seconds: 112.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 729 (time 1781231291.13 total 10.00 last 72.88) - open/close 1599 (time 1781231301.13 total 20.01 last 86.97) - open/close 2474 (time 1781231311.14 total 30.01 last 87.49) - open/close 3265 (time 1781231321.14 total 40.01 last 79.08) - open/close 3992 (time 1781231331.16 total 50.03 last 72.53) total: 4500 open/close in 56.20 seconds: 80.07 ops/second - /unlink 1286 (time 1781231351.93 total 10.02 last 128.36) - /unlink 2512 (time 1781231361.93 total 20.02 last 122.52) - /unlink 4058 (time 1781231371.94 total 30.03 last 154.57) total: 4500 /unlink in 33.20 seconds: 135.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 726 (time 1781231390.99 total 10.01 last 72.55) - open/close 1535 (time 1781231400.99 total 20.01 last 80.84) - open/close 2398 (time 1781231411.00 total 30.02 last 86.23) - open/close 3108 (time 1781231421.01 total 40.03 last 70.92) - open/close 3788 (time 1781231431.02 total 50.04 last 67.95) - open/close 4483 (time 1781231441.03 total 60.05 last 69.43) total: 4500 open/close in 60.23 seconds: 74.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1282 (time 1781231455.26 total 10.00 last 128.17) - /unlink 2508 (time 1781231465.27 total 20.01 last 122.53) - /unlink 3705 (time 1781231475.27 total 30.01 last 119.62) total: 4500 /unlink in 35.48 seconds: 126.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 706 (time 1781231495.02 total 10.01 last 70.54) - open/close 1278 (time 1781231505.04 total 20.03 last 57.08) - open/close 1865 (time 1781231515.05 total 30.03 last 58.68) - open/close 2561 (time 1781231525.06 total 40.04 last 69.54) - open/close 3229 (time 1781231535.07 total 50.05 last 66.73) - open/close 3812 (time 1781231545.07 total 60.06 last 58.30) total: 4500 open/close in 68.55 seconds: 65.65 ops/second - /unlink 1379 (time 1781231568.58 total 10.00 last 137.89) - /unlink 2819 (time 1781231578.58 total 20.01 last 143.93) - /unlink 4185 (time 1781231588.60 total 30.02 last 136.41) total: 4500 /unlink in 32.68 seconds: 137.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 759 (time 1781231607.53 total 10.00 last 75.86) - open/close 1372 (time 1781231617.53 total 20.01 last 61.26) - open/close 2067 (time 1781231627.54 total 30.02 last 69.46) - open/close 2733 (time 1781231637.54 total 40.02 last 66.56) - open/close 3393 (time 1781231647.55 total 50.03 last 65.98) - open/close 3962 (time 1781231657.60 total 60.08 last 56.59) - open/close 4467 (time 1781231667.61 total 70.09 last 50.47) total: 4500 open/close in 70.54 seconds: 63.79 ops/second - /unlink 1101 (time 1781231683.96 total 10.00 last 110.07) - /unlink 2296 (time 1781231693.97 total 20.01 last 119.41) - /unlink 3358 (time 1781231703.98 total 30.01 last 106.15) total: 4500 /unlink in 39.73 seconds: 113.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 697 (time 1781231727.76 total 10.01 last 69.66) - open/close 1430 (time 1781231737.76 total 20.01 last 73.28) - open/close 2197 (time 1781231747.77 total 30.02 last 76.62) - open/close 2957 (time 1781231757.77 total 40.02 last 75.99) - open/close 3714 (time 1781231767.77 total 50.02 last 75.70) - open/close 4395 (time 1781231777.78 total 60.02 last 68.07) total: 4500 open/close in 61.62 seconds: 73.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1286 (time 1781231794.68 total 10.01 last 128.52) - /unlink 2703 (time 1781231804.68 total 20.01 last 141.68) - /unlink 3996 (time 1781231814.69 total 30.01 last 129.22) total: 4500 /unlink in 33.71 seconds: 133.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 648 (time 1781231834.01 total 10.01 last 64.74) - open/close 1310 (time 1781231844.01 total 20.01 last 66.18) - open/close 1921 (time 1781231854.02 total 30.02 last 61.08) - open/close 2663 (time 1781231864.02 total 40.02 last 74.17) - open/close 3326 (time 1781231874.02 total 50.02 last 66.28) - open/close 3949 (time 1781231884.03 total 60.03 last 62.24) total: 4500 open/close in 68.67 seconds: 65.53 ops/second - /unlink 1246 (time 1781231907.50 total 10.00 last 124.54) - /unlink 2407 (time 1781231917.51 total 20.01 last 116.08) - /unlink 3683 (time 1781231927.52 total 30.02 last 127.44) total: 4500 /unlink in 40.00 seconds: 112.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 742 (time 1781231953.92 total 10.01 last 74.16) - open/close 1378 (time 1781231963.92 total 20.01 last 63.60) - open/close 2106 (time 1781231973.93 total 30.02 last 72.72) - open/close 2882 (time 1781231983.93 total 40.02 last 77.54) - open/close 3411 (time 1781231993.94 total 50.03 last 52.87) - open/close 3962 (time 1781232003.94 total 60.03 last 55.08) total: 4500 open/close in 67.50 seconds: 66.67 ops/second - /unlink 1261 (time 1781232026.54 total 10.00 last 126.06) - /unlink 2571 (time 1781232036.54 total 20.00 last 130.98) - /unlink 3912 (time 1781232046.55 total 30.01 last 133.98) total: 4500 /unlink in 34.61 seconds: 130.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 557 (time 1781232067.29 total 10.03 last 55.55) - open/close 1165 (time 1781232077.29 total 20.03 last 60.79) - open/close 1926 (time 1781232087.30 total 30.03 last 76.08) - open/close 2606 (time 1781232097.30 total 40.03 last 67.97) - open/close 3227 (time 1781232107.31 total 50.05 last 62.03) - open/close 4025 (time 1781232117.33 total 60.07 last 79.62) total: 4500 open/close in 66.45 seconds: 67.72 ops/second - /unlink 1103 (time 1781232138.89 total 10.01 last 110.21) - /unlink 2532 (time 1781232148.89 total 20.01 last 142.85) - /unlink 3895 (time 1781232158.89 total 30.02 last 136.23) total: 4500 /unlink in 35.07 seconds: 128.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 706 (time 1781232179.00 total 10.00 last 70.60) - open/close 1388 (time 1781232189.01 total 20.01 last 68.13) - open/close 2116 (time 1781232199.01 total 30.01 last 72.79) - open/close 2813 (time 1781232209.02 total 40.02 last 69.64) - open/close 3563 (time 1781232219.02 total 50.03 last 74.97) - open/close 4177 (time 1781232229.03 total 60.03 last 61.37) total: 4500 open/close in 64.15 seconds: 70.14 ops/second - /unlink 1249 (time 1781232248.17 total 10.00 last 124.89) - /unlink 2593 (time 1781232258.17 total 20.00 last 134.40) - /unlink 3766 (time 1781232268.17 total 30.00 last 117.29) total: 4500 /unlink in 37.00 seconds: 121.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 673 (time 1781232290.75 total 10.00 last 67.29) - open/close 1439 (time 1781232300.75 total 20.00 last 76.57) - open/close 2235 (time 1781232310.76 total 30.01 last 79.53) - open/close 2965 (time 1781232320.78 total 40.03 last 72.85) - open/close 3724 (time 1781232330.79 total 50.04 last 75.87) - open/close 4487 (time 1781232340.79 total 60.04 last 76.30) total: 4500 open/close in 60.23 seconds: 74.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1581 (time 1781232354.94 total 10.00 last 158.10) - /unlink 2884 (time 1781232364.94 total 20.00 last 130.25) total: 4500 /unlink in 29.80 seconds: 151.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 756 (time 1781232389.50 total 10.01 last 75.49) - open/close 1537 (time 1781232399.50 total 20.02 last 78.07) - open/close 2367 (time 1781232409.51 total 30.02 last 82.99) - open/close 3056 (time 1781232419.54 total 40.05 last 68.67) - open/close 3752 (time 1781232429.56 total 50.07 last 69.48) total: 4500 open/close in 58.15 seconds: 77.39 ops/second - /unlink 1249 (time 1781232452.49 total 10.00 last 124.84) - /unlink 2518 (time 1781232462.49 total 20.01 last 126.82) - /unlink 3960 (time 1781232472.50 total 30.01 last 144.15) total: 4500 /unlink in 33.44 seconds: 134.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 832 (time 1781232492.38 total 10.00 last 83.16) - open/close 1589 (time 1781232502.40 total 20.03 last 75.54) - open/close 2401 (time 1781232512.41 total 30.03 last 81.15) - open/close 3347 (time 1781232522.42 total 40.04 last 94.49) - open/close 4196 (time 1781232532.43 total 50.05 last 84.85) total: 4500 open/close in 54.01 seconds: 83.31 ops/second - /unlink 1749 (time 1781232550.91 total 10.00 last 174.87) - /unlink 3341 (time 1781232560.92 total 20.00 last 159.15) total: 4500 /unlink in 28.38 seconds: 158.59 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94088 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded.