It is time.
As a Java developer, this probably should be difficult to say, but it’s not. Now is the time to disable the Java browser plugin by default. In the past year I’ve only consciously used it on one site, and in that time there have been a whole bunch of security vulnerabilities in it.
The era of native plugins is nearly over, Java included. There are still a few sites out there which need it, but those are so few and far between that I’d recommend keeping it installed and disabling the plugin and re-enabling only when absolutely required.
This is trivial to do in Firefox, and I’m sure must be in other browsers as well.