8000 bloop/notes/v1.3.0.md at master · jiminhsieh/bloop · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
384 lines (289 loc) · 19.1 KB

v1.3.0.md

File metadata and controls

384 lines (289 loc) · 19.1 KB

bloop v1.3.0

Bloop is a build server and CLI tool for the Scala programming language developed by the Scala Center.

Bloop v1.3.0 is the biggest release of this project to this date.

With over 520 commits since the v1.2.5 release, ~26000 additions and ~9500 deletions, it aggregates together fundamental changes in the compilation pipeline, novel build server features, user-facing improvements and bug fixes together.

Upgrade guide 📥

These are only the upgrade steps. If you don't have bloop installed, you must read the installation instructions instead.

If you're on macOS, upgrade to the latest version with:

$ brew upgrade scalacenter/bloop/bloop
$ brew services restart bloop # Note a restart is required in case a server still runs!

If you're on Windows using scoop, upgrade to the latest version with:

$ scoop upgrade bloop
$ bloop ng-stop
$ bloop server # in a long-running terminal session

🆕 If you're on Arch Linux, upgrade to the latest version with:

$ rm -rf ~/.bloop
$ yaourt -S bloop
$ systemctl --user start bloop

Otherwise, use the generic installation method:

$ curl -L https://github.com/scalacenter/bloop/releases/download/v1.3.0/install.py | python
$ bloop ng-stop
$ systemctl --user start bloop # or similar way to run server in a long-running session

Read the complete installation instructions in our Installation page.

New compilation engine with client guarantees 💪 👪

v1.3.0 features a redesigned compilation engine built to solve tooling challenges already present in our current developer workflow but brought to the foreground by the formalized use of a client-server architecture. This compilation engine sports a more efficient way of doing IO work (with no compromises in correctness) and provides a new contract to build clients.

Bloop v1.3.0 and onwards guarantees clients that their actions will have the smallest usage footprint possible and will not conflict with those of other concurrent clients being served by the same server in the same build.

For example, if Metals is compiling your project via Bloop and you spawn a bloop CLI command such as bloop test foo --watch at the same time, Bloop guarantees that:

  1. The server heavily caches compilations for the same inputs (aka compile deduplication)

    If inputs haven't changed between the requests, only the first client request will trigger a compilation. The compilation of the second client will be deduplicated based on the compilation side effects recorded by the build server, so only one compilation will happen.

  2. Different compilation requests in the same build can run concurrently (aka compile isolation)

    If inputs have changed between requests, Bloop will compile the changed projects concurrently, avoiding shared state and conflicts with ongoing compilations.

  3. The outputs produced by both requests are independent in the file system.

    The compilation products will be stored in independent target directories only owned by the client. This independence is essential to allow clients to independently run any build action without altering task executions. Think avoiding crashing JVMs because other processes are writing contents to open class file pointers!

This is the first step towards modelling these build actions as pure functions despite their pervasive side effects. Bloop manages any internal state as best as possible to provide the best developer experience to end users. There will be more work in this area in the future!

Related pull requests

  1. Isolate and deduplicate compilation and Zipkin tracing support by jvican
  2. Pre-populate client classes dir if empty by jvican
  3. Deduplicate compilation if project inputs change by jvican
  4. Invalidate symbols with forked and local javac by jvican
  5. Don't create and delete empty classes dirs by jvican
  6. Cancel deduplication if ongoing compilation looks dead by jvican
  7. Use generic classes dir for the pipelined traversal by jvican
  8. Log to the user a deduplication is happening by jvican
  9. Revalidate class files compiled by dependent projects by jvican
  10. Fix not found type errors during incremental compilation by jvican
  11. Add CLI lock timeout and be more flexible when locking CLI clients
  12. Fix remaining issues around handling of compile products by jvican
  13. Catch exceptions when computing the bundle by jvican

New Zipkin tracing support 📡

Bloop v1.3.0 traces any compilation request to Zipkin if a Zipkin server is running in the background. Here's a screenshots showing collected compilation traces for the akka/akka project.

A Zipkin instance with build traces from the akka/akka project

And another screenshot showing the top part of a detailed compilation trace.

A Zipkin instance showing a detailed profile of a build trace from the akka/akka project

The zipkin integration is explained in the performance guide below.

In-depth compiler performance guide 🚀

The compiler performance guide is a new addition to our documentation that explains users how to:

  1. Get familiar with the performance characteristics of project builds
  2. Compare the performance of Bloop with that of other tools compiling Scala code
  3. Modify a complicated build to optimize build times

The guide is rich in details and written to provide pointers to those users wanting to speed up their workflow with Bloop and optimize their current build times further.

It is open for any community addition, so if you'd like to add something, help us improve it!

Faster build pipelining 🏇

Build pipelining is a compilation performance technique to start the compilation of modules in a build before their dependencies have finished compilation.

Build pipelining was merged back in December, 2018 and made available in bloop v1.2.1, making bloop the first Scala developer tool to implement this technique and support it under an experimental flag.

Bloop v1.3.0 changes its internal implementation to make it faster. We're very excited about what this entails for any Scala build in the wild. The bloop project will publish benchmarks and a concise guide in a follow-up post, keep tuned! 😄

Use Ammonite by default in bloop console 🎮

Contributed by @torkelrogstad.

Ammonite is a modernized Scala REPL with syntax highlighting, multi-line editing, the ability to load maven artifacts directly in the REPL, and many other quality-of-life improvements missing in the default Scala REPL.

Bloop v1.3.0 defaults on Ammonite every time you run bloop console on a project. Unlike running amm in your terminal, the Ammonite integration will automatically start an Ammonite REPL using the classpath of a project in your build. That allows you to use all the code defined in that classpath and experiment with its behavior.

Interested in the implementation details? Have a look at Add support for Ammonite REPL by default.

Better Metals and BSP integration 🤘 💯

BSP support in bloop v1.3.0 is vastly superior to that in v1.2.5. The new release implements a few non-implemented endpoints, fixes several non-reported issues in diagnostics and improves connection support and cancellation capabilities.

The Metals integration is superior to that in v1.2.5 thanks to this non-exhaustive list of PRs:

  1. Guarantee cancellation upon sudden BSP client dropouts by jvican
  2. Upgrade to BSP 2.0.0-M4 by tgodzik
  3. Implement BSP buildTarget/scalaMainClasses by marek1840
  4. Implement BSP buildTarget/scalaTestClasses by marek1840
  5. Send out server BSP responses/notifications in order by jvican
  6. A few unlisted PRs that casually fix issues in BSP semantics

Add support for Scala 2.13 ▶️

Bloop v1.3.0 finally adds support for Scala 2.13.x.

Add support for JDK 9 and 11 ⏩

Contributed by @er1c, bloop v1.3.0 supports JDK 9 and JDK 11. If you run bloop with these Java runtimes, it is recommended to enable the full parallel GC with -XX:+UseParallelGC as compilation is faster.

For this reason, Java 8 still remains the recommended runtime for the time being and our installers might install it by default. In the future, we might consider upgrading to a newer Java runtime.

  1. Compile on Java9+ by er1c
  2. Add Java9+ reverse-compatible changes by er1c

New Arch Linux installer 🏄

Contributed by TheElectronWill.

Bloop has a new AUR package for Arch Linux that is updated by our release infrastructure. You can now install bloop in Arch Linux by running pacaur -S bloop or yaourt -S bloop.

Improved file watcher responsiveness ⌚ 🔥

bloop v1.3.0 has better and snappier file watcher infrastructure, with protection against overflowing events and several fixes to bugs reported after v1.2.5. One of these bugs is a recurrent problems in macOS machines where Bloop server's JVM would crash due to uncleaned state in the Carbon-based file watcher.

Support for file watching in run commands has not yet been implemented. If you want to help, drop by bloop's Gitter channel and we will give you pointers to help implement it.

Fix use of test arguments in bloop test 📈

Running bloop test in your project or passing specific test arguments to Bloop's test runner could produce issues such as this one:

[E] Unexpected error during remote test execution: 'Socket closed'.
[E] Internal error when running tests: sbt.ForkMain$Run$RunAborted: java.net.SocketException: Broken pipe (Write failed)

This pull request makes sure that we handling missing test frameworks better and that the above issues processing test arguments are gone.

Improvements in build tools support (bloopInstall) 🛠️

Improvements in sbt

Automatic support for integration tests in sbt builds

sbt-bloop v1.3.0 will automatically export integration test configurations to independent build targets. If you have modified your sbt build to support integration tests in your build, you can safely remove that logic.

Support sbt source dependencies by default

Contributed by olafurpg. Support for sbt project references was before hidden behind an specific sbt boolean setting. Now it's enabled by default.

Other

  1. Avoid duplication of projects in dependencies field by tkroman
  2. Fix small bug when exporting scalac and javac options in sbt by jvican

Improvements in Gradle

  1. Include Gradle's transitive dependencies in classpath by Arthurm1
  2. Ignore annotation processor in Gradle by tgodzik
  3. Don't add JAR dependencies of Gradle projects in the classpath by tgodzik
  4. Export source dependencies from Gradle by tgodzik

Thanks to these improvements, Metals will be able to support "Import Build" in Gradle projects in its next release.

Improvements in Mill

  1. Point Mill docs to merged bloop integration upstream by Baccata
  2. Fix broken link in Mill documentation by perryao
  3. Update Mill version by robby-phd

Thanks to these improvements, Metals will be able to support "Import Build" in mill builds in its next release.

Improvements in Maven

  1. Avoid NPE when collecting resources in Maven's Bloop plugin by artyomboyko
  2. Add source dependencies to Maven plugin by tgodzik
  3. Fix warning about Maven being run in interactive mode by tgodzik
  4. Fix Maven plugin in Windows by adding version of maven-invoker by tgodzik

Thanks to these improvements, Metals will be able to support "Import Build" in Maven projects in its next release.

Miscellaneous

Here's a list of key miscellaneous changes in this release.

  1. Synchronize CLI clients to the same build by jvican
  2. Don't recompile project if server dies by jvican
  3. Add -Dbloop.computation.cores and env var to configure thread pool by jvican
  4. Display Java version and Java home in bloop about by cquiroz
  5. Add failing test case where source files are pulled in multiple times by jackkoenig
  6. Don't print warnings when autocompleting by CucumisSativus
  7. Add buildpress, a tool to automatically export from any build tool to bloop by jvican
  8. Add support for standalone repo path in CLI args by tkroman
  9. Add contributing guide to the website by jvican
  10. Improve bloop launcher's test suite by jvican
  11. Change SCOOP env variable to avoid conflict by Arthurm1
  12. Show correct default for install.py '--dest' option by BardurArantsson
  13. Use a guaranteed unique name for temporary directories by propensive
  14. Update website, enable versioning and assort fixes by jvican
  15. Update contributing guide with expected error by CucumisSativus
  16. Fix installation link in README by jrudolph
  17. Encode UUIDs with Base 64 by odisseus
  18. Fix off-one error in fish autocompletions by 2m
  19. Use two cores for Bloop's computation pool in Travis by jvican
  20. Update docs URL in Bloop error messages by AckoLF
  21. Update Homebrew server log path by regadas

Next steps 🚶

The recent work on supporting concurrent build clients, giving strong building guarantees to clients and the bloop launcher released in v1.2.5 has crystallized in a robust tooling framework to take our build tool integrations to the next level.

Bloop v1.4.0 will enable sbt to offload compilation of projects automatically to Bloop, speeding up sbt project load and making the cutting-edge performance improvements in Bloop available to sbt. The goal is that users will be able to use bloop without changing their current development workflow.

Such integrations for other build tools will also be possible. The bloop team volunteers to mentor people interested in contributing such integrations for popular build tools such as mill or Gradle. If you're interested, please get in contact with us.

Special acknowledgements 🙏

bloop v1.3.0 is an exciting release because of the vast amount of external contributors that we have had. Since v1.2.5, bloop has received contributions by more than 27 OSS contributors.

Additionally, the bloop team has welcomed three new more members: Tomasz Godzik (@tgodzik) and Marek Żarnowski (@marek1840) from VirtusLab which have made this release a reality and Roman Tkalenko (@tkroman) which has recently started contributing in making Bloop better for CI workflows.

I am thrilled to see the bloop community grow and be working with such skilled people in improving the tooling in the Scala community.

According to git shortlog -sn --no-merges v1.2.5..v1.3.0, 27 people contributed to this release: Guillaume R, Roman Tkalenko, Jorge, Eric Peters, Michał Gutowski, Marek Żarnowski, Carlos Quiroz, Artyom Boyko, Arthur McGibbon, Tomasz Godzik, Torkel Rogstad, Myroslav Golub, AckoLF, Ólafur Páll Geirsson, ArthurM, Bardur Arantsson, David Gregory, Enver Osmanov, Filipe Regadas, Johannes Rudolph, Jon Pretty, Martynas Mickevičius, Mike, Olivier Mélois, Jack Koenig and Robby.

0