Wait, to be clear: these three attributes should only be added to the .jar file with class files and not to the .jar files with sources and with javadocs?
Somewhat off-topic: does this change mean that I should build future releases in a Debian stable VM, not in macOS, for maximum reproducibility?
Wait, to be clear: these three attributes should only be added to the .jar file with class files and not to the .jar files with sources and with javadocs?
It's more important for the class-jar, but if you think it is more consistent, just add them all over.
Somewhat off-topic: does this change mean that I should build future releases in a Debian stable VM, not in macOS, for maximum reproducibility?
No not at all. I think building it anywhere given the correct jdk and dependencies ought to be fine. This information could just be helpful when people build elsewhere and then report bugs (cf. around 19720#comment:30). This manifest entry would give us a hint where things were running.
Hmmmm, I think I agree with the changes above, but would you mind preparing a patch that I can review and try out? I'm worried that I'll otherwise incorporate those changes in another way than intended. (Thanks!)
Ah, I didn't mean only the checkstyle parts but also the build.xml parts above. Anyway, let's focus on checkstyle for the moment.
Looking at the checkstyle commits I noticed that even those two don't contain the exact same output. And we're still missing three more (Onionoo, Metrics website, and ExoneraTor). How about we work on one checkstyle file for one code base, and once that's good, we copy that over to the others?
Quick question: Why add two modules SuppressWarningsFilter and SuppressWarningsHolder? Those are not explained in the changes section. Do we need both, and why?
The remaining parts that don't make immediate sense are:
Why add two modules SuppressWarningsFilter and SuppressWarningsHolder? Those are not explained in the changes section. Do we need both, and why?
Can you prepare a patch for, say, metrics-lib's metrics_checks.xml as complete version with all changes and documentation why we're making those changes and which checkstyle issues are affected in brackets, that we can use to discuss and later copy over to the other four code bases?
Can you submit a patch for metrics-lib's build.xml with the various suggestions in comments 2 to 6 above? I'm afraid I lost track which of them should go where.
I'd propose having another git repository metrics-base or metrics-java-base for providing the standard build elements, which currently would be a base-build.xml and metrics' checkstyle rules.
This can be referenced as a sub-module and thus avoid code duplication.
I'd propose having another git repository metrics-base or metrics-java-base for providing the standard build elements, which currently would be a base-build.xml and metrics' checkstyle rules.
This can be referenced as a sub-module and thus avoid code duplication.
Plausible!
Thoughts?
First thought is that we should try to avoid confusing new contributors. Basically, we're adding yet another hurdle for them to get up to speed. Though I admit that this hurdle is worth considering to avoid duplicating code.
What we'd have to do is fail with a useful error message in case this submodule has not yet been initialized. That message should contain the exact command that is necessary to initialize the missing submodule. Wording is important here, I think.
Another thought is that we should make sure that this won't break any future Jenkins tasks. I believe initializing a Git submodule from the same repository is okay, but we should check. Copying hiro on this ticket for that reason.
I'd propose having another git repository metrics-base or metrics-java-base for providing the standard build elements, which currently would be a base-build.xml and metrics' checkstyle rules.
This can be referenced as a sub-module and thus avoid code duplication.
Plausible!
Which name?
metrics-base or metrics-java-base or something else?
Thoughts?
First thought is that we should try to avoid confusing new contributors. Basically, we're adding yet another hurdle for them to get up to speed. Though I admit that this hurdle is worth considering to avoid duplicating code.
What we'd have to do is fail with a useful error message in case this submodule has not yet been initialized. That message should contain the exact command that is necessary to initialize the missing submodule. Wording is important here, I think.
Yes, the error message should be clear.
Actually the submodule update could be perfomed by ant.
Another thought is that we should make sure that this won't break any future Jenkins tasks.
The submodule setup only needs ant and git, which are available in a Jenkins build.
I'd propose having another git repository metrics-base or metrics-java-base for providing the standard build elements, which currently would be a base-build.xml and metrics' checkstyle rules.
This can be referenced as a sub-module and thus avoid code duplication.
Plausible!
Which name?
metrics-base or metrics-java-base or something else?
Hmm, how about metrics-base and we add any shared resources regardless of language? In the beginning there'd be only Java files, but would it hurt if we'd add files that are only relevant for other languages?
Thoughts?
First thought is that we should try to avoid confusing new contributors. Basically, we're adding yet another hurdle for them to get up to speed. Though I admit that this hurdle is worth considering to avoid duplicating code.
What we'd have to do is fail with a useful error message in case this submodule has not yet been initialized. That message should contain the exact command that is necessary to initialize the missing submodule. Wording is important here, I think.
Yes, the error message should be clear.
Actually the submodule update could be perfomed by ant.
Sounds good!
Another thought is that we should make sure that this won't break any future Jenkins tasks.
The submodule setup only needs ant and git, which are available in a Jenkins build.
Including the network access to fetch the dependent repository from Tor's Git server? If so, great!
..
Hmm, how about metrics-base and we add any shared resources regardless of language? In the beginning there'd be only Java files, but would it hurt if we'd add files that are only relevant for other languages?
Yes, metrics-base is the best choice. And, sure, should other languages be included when needed.
How to proceed? I'll continue with the streamlining. Who will request the new repo etc.?
Yes, metrics-base is the best choice. And, sure, should other languages be included when needed.
How to proceed? I'll continue with the streamlining. Who will request the new repo etc.?
There, I just created #21040 (moved) for the repository.
Although our projects seem generic, there are quite some more or less subtle differences.
I guess, I generated the common part for metrics-base now. It should be sufficient to fill the repo and a non-empty one is necessary for initializing the git sub-modules correctly. The other changes will follow once metrics-base contains this patch.
The bootstrapping by executing the correct git commands will have to be performed one time and after that an ant task will update the submodule without user interference.
Please review the changes in:
Onionoo one commit,
metrics-lib (three commits, the first unrelated test problems, the third a suggestion for documentation),
CollecTor also only one commit.
Before that the attached patch for metrics-base needs to be applied.
Test steps:
clone using --recursive; if checkout happened without that use src/main/resources/bootstrap-development.sh.
metrics-web and exonerator are not included yet, b/c those are not yet released and quite different.
Whee, quite some code to review here. Sure, a lot of code has just moved around, but that only makes it trickier to spot the little differences. I finished a first (tool-supported) read of the changes but didn't finish trying it all out. But hopefully this initial feedback is useful anyway:
The two properties "libs" and "source-and-target-java-version" are listed twice.
Speaking of "source-and-target-java-version", we should probably keep that property project-specific. Otherwise we'll have to raise the Java version for all projects at once, including metrics-lib and projects depending on metrics-lib.
The Checkstyle configuration file java/metrics_checks.xml does not contain modules SuppressWarningsFilter and SuppressWarningsHolder that we added in metrics-lib's version nor module UnusedImports in Onionoo's version. It also still defines a maximum line length of 100 rather than 80. Let's combine the latest settings from all three code bases here.
This is certainly a minor issue, but the usage instructions in the usage target should all use the same capitalization and sentence structure, like "'name' does XY", not "'name' Does XY" or "'name' XY".
It's already time to update the copyright year in the docs target! Happy 2017!
Why did you change the "jetty.version" property to "" rather than "-8.1.16.v20140903"?
The new generic tar target does not include the DESIGN file anymore. I'd argue that we should remove that file anyway, but if you want to keep it, we'll have to include it in the release tarball, too.
The script file src/main/resources/bootstrap-development.sh should be checked in with the executable flag.
This commit or a subsequent commit should remove src/test/resources/metrics_checks.xml which is now obsolete.
Which tests did not pass prior to this commit? They run through just fine here.
It shouldn't be necessary to initialize attributes with null or 0, which is the default anyway. All new no-args constructors should be empty (except for a comment saying that they're empty on purpose).
The script file src/main/resources/bootstrap-development.sh should be checked in with the executable flag.
This commit or a subsequent commit should remove src/test/resources/junittest.policy and src/test/resources/metrics_checks.xml which are now obsolete.
I guess I'll wait for the revision before trying out these changes altogether. My plan was to create release tarballs and compare them with the latest releases.
The two properties "libs" and "source-and-target-java-version" are listed twice.
Removed.
Speaking of "source-and-target-java-version", we should probably keep that property project-specific. Otherwise we'll have to raise the Java version for all projects at once, including metrics-lib and projects depending on metrics-lib.
If a project deviates from the base java version, only the property needs to be added to build.xml and takes precedence. (Try setting it to java 1.4 and see how the compile complains about generics.)
I added a comment to the build.template.xml
The Checkstyle configuration file java/metrics_checks.xml does not contain modules SuppressWarningsFilter and SuppressWarningsHolder that we added in metrics-lib's version nor module UnusedImports in Onionoo's version. It also still defines a maximum line length of 100 rather than 80. Let's combine the latest settings from all three code bases here.
Changed the line length check; all others were already part of the initial check-in.
This is certainly a minor issue, but the usage instructions in the usage target should all use the same capitalization and sentence structure, like "'name' does XY", not "'name' Does XY" or "'name' XY".
Streamlined.
It's already time to update the copyright year in the docs target! Happy 2017!
Good that this only needs a change in one place now :-)
The year is now computed from the current time UTC.
Why did you change the "jetty.version" property to "" rather than "-8.1.16.v20140903"?
Fixed.
The new generic tar target does not include the DESIGN file anymore. I'd argue that we should remove that file anyway, but if you want to keep it, we'll have to include it in the release tarball, too.
Removed. Designs should be elsewhere, for example Tech-Reports.
The script file src/main/resources/bootstrap-development.sh should be checked in with the executable flag.
Done.
This commit or a subsequent commit should remove src/test/resources/metrics_checks.xml which is now obsolete.
Which tests did not pass prior to this commit? They run through just fine here.
Hmm, now these tests pass here, too. I'll go hunting for the jdk/environment setting responsible, but this shouldn't affect this ticket. Please, just cherry pick.
It shouldn't be necessary to initialize attributes with null or 0, which is the default anyway. All new no-args constructors should be empty (except for a comment saying that they're empty on purpose).
The compiler complains about uninitialized fields (those are final in the respective classes). Otherwise, same as the previous.
The script file src/main/resources/bootstrap-development.sh should be checked in with the executable flag.
Done.
This commit or a subsequent commit should remove src/test/resources/junittest.policy and src/test/resources/metrics_checks.xml which are now obsolete.
Done.
I guess I'll wait for the revision before trying out these changes altogether. My plan was to create release tarballs and compare them with the latest releases.
Be aware, that Onionoo does not include the protocol version in the manifest anymore, because the protocol version is part of the release version now.
Okay, here's what I did and where I ran into issues:
Pushed your two metrics-base patches to metrics-base without changes.
Tried running ant tar in your onionoo task-20596-submod branch, but it keeps telling me that signing of at least one of the .jars failed. I didn't investigate further yet. Does that work for you?
Rebased your metrics-lib task-20596-submod branch and pushed it to branch task-20596 in my repository. (Though I'm planning to squash your squash commit into "Implements task-20596" rather than "Added development description" when merging to master.) Realized that Gson is indeed unhappy, so we should keep your commit, but without explicit null-initializations, if possible. However, I ran into two issues:
The sources jar contains sources in a different place than previous tarballs.
The executable jar contains other classes than just our own, which it shouldn't (as opposed to the other executable jars).
Rebased your collector task-20596-submod branch and pushed it to branch task-20596 in my repository. Building seems to work fine, but I found this warning: "[javadoc] javadoc: error - Error while reading file /Users/karsten/src/collector/src/main/resources/overview.html". I didn't compare .jar file contents yet but expect to find similar differences as I found in the metrics-lib .jar files. Same goes for Onionoo above.
Would you want to look into some of these issues? I can do another round of tests and checks tomorrow. Thanks!
Okay, here's what I did and where I ran into issues:
Pushed your two metrics-base patches to metrics-base without changes.
Tried running ant tar in your onionoo task-20596-submod branch, but it keeps telling me that signing of at least one of the .jars failed. I didn't investigate further yet. Does that work for you?
That worked fine in the test setting I had, well. I'll look into it.
Rebased your metrics-lib task-20596-submod branch and pushed it to branch task-20596 in my repository. (Though I'm planning to squash your squash commit into "Implements task-20596" rather than "Added development description" when merging to master.) Realized that Gson is indeed unhappy, so we should keep your commit, but without explicit null-initializations, if possible. However, I ran into two issues:
The sources jar contains sources in a different place than previous tarballs.
The executable jar contains other classes than just our own, which it shouldn't (as opposed to the other executable jars).
Rebased your collector task-20596-submod branch and pushed it to branch task-20596 in my repository. Building seems to work fine, but I found this warning: "[javadoc] javadoc: error - Error while reading file /Users/karsten/src/collector/src/main/resources/overview.html". I didn't compare .jar file contents yet but expect to find similar differences as I found in the metrics-lib .jar files. Same goes for Onionoo above.
Would you want to look into some of these issues? I can do another round of tests and checks tomorrow. Thanks!
No problem, I'll take a look at all the issues. Thanks for checking quickly!
Could you specify the source location differences?
And, (without knowing how weird the paths might be) is the new path structure problematic?
The old sources .jar file contained paths like org/torproject/descriptor/BandwidthHistory.java, whereas the new file contains that file in main/java/org/torproject/descriptor/BandwidthHistory.java. The new file also contains other sources and resources that the old file did not contain. I'm not using sources .jar files anywhere, but I could imagine that some tools depend on the old directory structure, so it might be good to keep it. But I don't know for sure.
All things should be resolved now except for the javadoc complaint about overview.html.
This doesn't hinder javadoc creation so fine for this ticket. But, as we ship javadoc with all products it would be a better style to provide an overview page with useful contents for each.
New ticket?
Pushed that fix to the official metrics-base repository.
Onionoo works fine now, except for one tiny change: The MANIFEST.MF file in the .jar file now says Created-By: 1.8.0_102-b14 (Oracle Corporation) rather than Created-By: The Tor Project. The .war file still says the latter. Any ideas?
Note that metrics-lib and CollecTor still don't work. See the problem above.
The JDK version was supposed to be included in the manifest (making reproducing builds easier). Thus, I added Implemented-By: The Tor Project and left Created-By: ... to refer to the JDK as it usually does. Hope, this is fine.
Other changes: make all three tar tasks work and only add Main-Class to the appropriate executable jar.
I tested by brute-force sub-repo replacement and it worked.
Neither CollecTor's *-sources.jar file nor its *-javadoc.jar file should contain the two files collector.properties and logback.xml. Can you remove those (but leave them in the main *.jar)?
Onionoo's .war file still contains manifest entry Created-By: The Tor Project which I guess should also be Implemented-By: The Tor Project as in the .jar file.
There should be change log entries in all three code bases on medium or minor level saying something like: "Replaced several build files or parts thereof with their equivalents from the metrics-base repository to unify the build process among metrics code bases." Feel free to tweak that sentence. I can add it while merging.
I just noticed that you use your own branches for collector and metrics-lib and assume there were no changes I should have looked at.
Remaining issues are:
Neither CollecTor's *-sources.jar file nor its *-javadoc.jar file should contain the two files collector.properties and logback.xml. Can you remove those (but leave them in the main *.jar)?
Fixed in metrics-base.
Onionoo's .war file still contains manifest entry Created-By: The Tor Project which I guess should also be Implemented-By: The Tor Project as in the .jar file.
Fixed in Onionoo.
There should be change log entries in all three code bases on medium or minor level saying something like: "Replaced several build files or parts thereof with their equivalents from the metrics-base repository to unify the build process among metrics code bases." Feel free to tweak that sentence. I can add it while merging.
Added changelog entries to onionoo, collector, descriptor ;-)
All changes on the known branches in my git repos.
We're getting closer! :)
Yes! This all might seem tedious, but while making the fixes it is already very much more efficient; and, I guess trying to keep all projects in sync without metrics-base would just cause exponentially more iterations if not being an impossible task in the long run.