== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:06:21 (1763337981) Stopping clients: oleg431-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg431-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg431-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg431-server oleg431-server: oleg431-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg431-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 644 (time 1763338157.79 total 10.00 last 64.40) - open/close 1300 (time 1763338167.80 total 20.01 last 65.56) - open/close 1974 (time 1763338177.80 total 30.01 last 67.39) - open/close 2644 (time 1763338187.80 total 40.01 last 66.99) - open/close 3385 (time 1763338197.81 total 50.02 last 74.01) - open/close 4114 (time 1763338207.82 total 60.03 last 72.84) total: 4500 open/close in 65.93 seconds: 68.25 ops/second - /unlink 1224 (time 1763338228.17 total 10.00 last 122.37) - /unlink 2467 (time 1763338238.17 total 20.00 last 124.28) - /unlink 3697 (time 1763338248.17 total 30.01 last 122.95) total: 4500 /unlink in 36.58 seconds: 123.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 676 (time 1763338268.53 total 10.01 last 67.54) - open/close 1403 (time 1763338278.54 total 20.02 last 72.63) - open/close 2100 (time 1763338288.55 total 30.03 last 69.63) - open/close 2778 (time 1763338298.56 total 40.04 last 67.73) - open/close 3478 (time 1763338308.56 total 50.04 last 69.98) - open/close 4150 (time 1763338318.56 total 60.05 last 67.17) total: 4500 open/close in 64.73 seconds: 69.52 ops/second - /unlink 1338 (time 1763338337.39 total 10.00 last 133.77) - /unlink 2556 (time 1763338347.39 total 20.00 last 121.79) - /unlink 3702 (time 1763338357.40 total 30.01 last 114.56) total: 4500 /unlink in 36.97 seconds: 121.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 667 (time 1763338378.82 total 10.00 last 66.69) - open/close 1251 (time 1763338388.84 total 20.02 last 58.32) - open/close 1925 (time 1763338398.85 total 30.03 last 67.31) - open/close 2578 (time 1763338408.85 total 40.03 last 65.28) - open/close 3260 (time 1763338418.86 total 50.04 last 68.13) - open/close 3933 (time 1763338428.86 total 60.04 last 67.29) total: 4500 open/close in 68.97 seconds: 65.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1150 (time 1763338452.75 total 10.01 last 114.91) - /unlink 2351 (time 1763338462.75 total 20.01 last 120.09) - /unlink 3626 (time 1763338472.75 total 30.01 last 127.46) total: 4500 /unlink in 37.00 seconds: 121.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 600 (time 1763338494.57 total 10.02 last 59.91) - open/close 1202 (time 1763338504.59 total 20.03 last 60.09) - open/close 1853 (time 1763338514.59 total 30.04 last 65.08) - open/close 2494 (time 1763338524.59 total 40.04 last 64.07) - open/close 3252 (time 1763338534.60 total 50.05 last 75.75) - open/close 3938 (time 1763338544.61 total 60.06 last 68.53) total: 4500 open/close in 68.16 seconds: 66.02 ops/second - /unlink 1256 (time 1763338566.62 total 10.01 last 125.51) - /unlink 2530 (time 1763338576.62 total 20.01 last 127.32) - /unlink 3782 (time 1763338586.63 total 30.02 last 125.17) total: 4500 /unlink in 36.04 seconds: 124.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 561 (time 1763338607.07 total 10.01 last 56.04) - open/close 1223 (time 1763338617.08 total 20.01 last 66.19) - open/close 1873 (time 1763338627.08 total 30.02 last 64.97) - open/close 2544 (time 1763338637.09 total 40.03 last 67.04) - open/close 3237 (time 1763338647.10 total 50.04 last 69.23) - open/close 3928 (time 1763338657.11 total 60.05 last 68.99) total: 4500 open/close in 68.54 seconds: 65.66 ops/second - /unlink 1177 (time 1763338680.12 total 10.00 last 117.64) - /unlink 2473 (time 1763338690.13 total 20.01 last 129.52) - /unlink 3791 (time 1763338700.13 total 30.01 last 131.78) total: 4500 /unlink in 35.75 seconds: 125.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 626 (time 1763338720.25 total 10.01 last 62.56) - open/close 1328 (time 1763338730.27 total 20.02 last 70.10) - open/close 1996 (time 1763338740.27 total 30.02 last 66.79) - open/close 2674 (time 1763338750.28 total 40.03 last 67.75) - open/close 3251 (time 1763338760.30 total 50.05 last 57.58) - open/close 3852 (time 1763338770.30 total 60.05 last 60.08) - open/close 4490 (time 1763338780.31 total 70.07 last 63.71) total: 4500 open/close in 70.26 seconds: 64.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1241 (time 1763338795.16 total 10.00 last 124.09) - /unlink 2495 (time 1763338805.16 total 20.00 last 125.38) - /unlink 3744 (time 1763338815.16 total 30.00 last 124.89) total: 4500 /unlink in 36.24 seconds: 124.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 694 (time 1763338835.75 total 10.01 last 69.35) - open/close 1393 (time 1763338845.76 total 20.01 last 69.87) - open/close 2033 (time 1763338855.77 total 30.03 last 63.90) - open/close 2729 (time 1763338865.77 total 40.03 last 69.59) - open/close 3388 (time 1763338875.79 total 50.05 last 65.78) - open/close 4015 (time 1763338885.80 total 60.06 last 62.63) total: 4500 open/close in 67.12 seconds: 67.05 ops/second - /unlink 1250 (time 1763338906.69 total 10.01 last 124.86) - /unlink 2559 (time 1763338916.70 total 20.02 last 130.80) - /unlink 3898 (time 1763338926.70 total 30.02 last 133.88) total: 4500 /unlink in 34.71 seconds: 129.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 697 (time 1763338945.95 total 10.00 last 69.68) - open/close 1308 (time 1763338955.96 total 20.02 last 61.02) - open/close 1957 (time 1763338965.97 total 30.02 last 64.88) - open/close 2546 (time 1763338975.97 total 40.03 last 58.86) - open/close 3249 (time 1763338985.97 total 50.03 last 70.29) - open/close 3998 (time 1763338995.99 total 60.04 last 74.79) total: 4500 open/close in 67.71 seconds: 66.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1386 (time 1763339017.49 total 10.01 last 138.48) - /unlink 2710 (time 1763339027.49 total 20.01 last 132.35) - /unlink 4024 (time 1763339037.50 total 30.02 last 131.36) total: 4500 /unlink in 33.54 seconds: 134.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 713 (time 1763339054.89 total 10.01 last 71.26) - open/close 1411 (time 1763339064.90 total 20.01 last 69.74) - open/close 2159 (time 1763339074.92 total 30.04 last 74.63) - open/close 2916 (time 1763339084.94 total 40.05 last 75.60) - open/close 3679 (time 1763339094.95 total 50.06 last 76.23) - open/close 4418 (time 1763339104.96 total 60.07 last 73.83) total: 4500 open/close in 61.20 seconds: 73.53 ops/second - /unlink 1255 (time 1763339119.48 total 10.00 last 125.45) - /unlink 2611 (time 1763339129.49 total 20.01 last 135.56) - /unlink 3993 (time 1763339139.49 total 30.01 last 138.13) total: 4500 /unlink in 33.79 seconds: 133.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 743 (time 1763339157.01 total 10.00 last 74.27) - open/close 1511 (time 1763339167.02 total 20.01 last 76.75) - open/close 2222 (time 1763339177.03 total 30.03 last 70.97) - open/close 2819 (time 1763339187.04 total 40.03 last 59.69) - open/close 3431 (time 1763339197.05 total 50.05 last 61.09) - open/close 3997 (time 1763339207.06 total 60.05 last 56.57) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1212 (time 1763339229.63 total 10.00 last 121.19) - /unlink 2444 (time 1763339239.63 total 20.00 last 123.18) - /unlink 3665 (time 1763339249.65 total 30.02 last 121.92) total: 4500 /unlink in 37.55 seconds: 119.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 673 (time 1763339272.50 total 10.01 last 67.24) - open/close 1403 (time 1763339282.51 total 20.01 last 72.97) - open/close 2158 (time 1763339292.51 total 30.02 last 75.46) - open/close 2868 (time 1763339302.52 total 40.03 last 70.95) - open/close 3592 (time 1763339312.54 total 50.04 last 72.28) - open/close 4305 (time 1763339322.54 total 60.05 last 71.25) total: 4500 open/close in 63.08 seconds: 71.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1304 (time 1763339339.64 total 10.01 last 130.28) - /unlink 2669 (time 1763339349.64 total 20.01 last 136.46) - /unlink 4099 (time 1763339359.65 total 30.01 last 142.96) total: 4500 /unlink in 32.96 seconds: 136.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 777 (time 1763339376.20 total 10.03 last 77.46) - open/close 1552 (time 1763339386.20 total 20.03 last 77.48) - open/close 2361 (time 1763339396.20 total 30.04 last 80.88) - open/close 3088 (time 1763339406.21 total 40.04 last 72.64) - open/close 3743 (time 1763339416.21 total 50.05 last 65.48) - open/close 4429 (time 1763339426.23 total 60.06 last 68.47) total: 4500 open/close in 60.98 seconds: 73.79 ops/second - /unlink 1390 (time 1763339441.62 total 10.00 last 138.98) - /unlink 2870 (time 1763339451.62 total 20.00 last 147.98) - /unlink 4348 (time 1763339461.62 total 30.00 last 147.80) total: 4500 /unlink in 31.00 seconds: 145.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 826 (time 1763339475.55 total 10.00 last 82.58) - open/close 1367 (time 1763339485.56 total 20.02 last 54.03) - open/close 1956 (time 1763339495.59 total 30.05 last 58.72) - open/close 2641 (time 1763339505.60 total 40.05 last 68.48) - open/close 3342 (time 1763339515.60 total 50.06 last 70.04) - open/close 4161 (time 1763339525.61 total 60.06 last 81.87) total: 4500 open/close in 64.30 seconds: 69.99 ops/second - /unlink 1304 (time 1763339543.49 total 10.00 last 130.40) - /unlink 2642 (time 1763339553.49 total 20.00 last 133.79) - /unlink 3877 (time 1763339563.49 total 30.00 last 123.48) total: 4500 /unlink in 34.80 seconds: 129.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 744 (time 1763339582.28 total 10.00 last 74.36) - open/close 1448 (time 1763339592.30 total 20.02 last 70.31) - open/close 2212 (time 1763339602.30 total 30.03 last 76.34) - open/close 3016 (time 1763339612.31 total 40.03 last 80.36) - open/close 3689 (time 1763339622.32 total 50.04 last 67.26) - open/close 4396 (time 1763339632.32 total 60.04 last 70.67) total: 4500 open/close in 61.43 seconds: 73.25 ops/second - /unlink 1500 (time 1763339647.56 total 10.00 last 149.99) - /unlink 3053 (time 1763339657.56 total 20.00 last 155.26) total: 4500 /unlink in 29.76 seconds: 151.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 690 (time 1763339681.23 total 10.00 last 69.00) - open/close 1359 (time 1763339691.24 total 20.01 last 66.85) - open/close 2081 (time 1763339701.24 total 30.01 last 72.18) - open/close 2811 (time 1763339711.26 total 40.02 last 72.90) - open/close 3470 (time 1763339721.27 total 50.03 last 65.84) - open/close 4134 (time 1763339731.28 total 60.05 last 66.31) total: 4500 open/close in 65.68 seconds: 68.51 ops/second - /unlink 1073 (time 1763339751.51 total 10.00 last 107.26) - /unlink 2402 (time 1763339761.51 total 20.00 last 132.89) - /unlink 3703 (time 1763339771.51 total 30.01 last 130.06) total: 4500 /unlink in 36.43 seconds: 123.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 794 (time 1763339791.21 total 10.01 last 79.28) - open/close 1456 (time 1763339801.21 total 20.02 last 66.14) - open/close 2246 (time 1763339811.22 total 30.03 last 78.96) - open/close 3055 (time 1763339821.23 total 40.03 last 80.86) - open/close 3827 (time 1763339831.24 total 50.04 last 77.13) total: 4500 open/close in 59.41 seconds: 75.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1357 (time 1763339854.72 total 10.01 last 135.62) - /unlink 2624 (time 1763339864.72 total 20.01 last 126.64) - /unlink 3961 (time 1763339874.73 total 30.01 last 133.66) total: 4500 /unlink in 34.27 seconds: 131.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 758 (time 1763339892.82 total 10.01 last 75.72) - open/close 1374 (time 1763339902.82 total 20.01 last 61.59) - open/close 2007 (time 1763339912.83 total 30.02 last 63.26) - open/close 2679 (time 1763339922.83 total 40.03 last 67.15) - open/close 3417 (time 1763339932.85 total 50.04 last 73.71) - open/close 4133 (time 1763339942.85 total 60.04 last 71.57) total: 4500 open/close in 65.04 seconds: 69.19 ops/second - /unlink 1323 (time 1763339961.62 total 10.01 last 132.20) - /unlink 2619 (time 1763339971.63 total 20.01 last 129.56) - /unlink 4019 (time 1763339981.63 total 30.02 last 139.90) total: 4500 /unlink in 33.63 seconds: 133.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 696 (time 1763339999.09 total 10.00 last 69.59) - open/close 1341 (time 1763340009.09 total 20.00 last 64.48) - open/close 2087 (time 1763340019.11 total 30.02 last 74.50) - open/close 2868 (time 1763340029.11 total 40.02 last 78.10) - open/close 3635 (time 1763340039.11 total 50.02 last 76.65) - open/close 4282 (time 1763340049.12 total 60.03 last 64.70) total: 4500 open/close in 63.37 seconds: 71.01 ops/second - /unlink 1340 (time 1763340066.98 total 10.00 last 133.98) - /unlink 2585 (time 1763340076.98 total 20.00 last 124.48) - /unlink 3775 (time 1763340086.99 total 30.01 last 118.96) total: 4500 /unlink in 36.67 seconds: 122.72 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93958 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.