We were able to have internet pretty much at all times when in Japan. Whether on a train, in a park, temple, or 100 yen store we could get to the internet. But the truth is there was almost no open free WiFi anywhere.
We rented a mobile WiFi hotspot, which connects through the mobile cell data networks- mostly LTE and dropping back to 3G when in lower coverage areas. It was probably one of the best things that we did for the trip- being able to do a google search or lookup directions on the go with google maps with our own devices at any time.
There were quite a few different companies that rented the devices, we ended up going with Telecom Square. Most of the places had kiosks/storefronts at the airport, was really easy picking it up and getting going. We were able to connect either up to 6 or 10 devices (I forget) and had pretty much unlimited data usage- the two weeks was around $150 or so. You can also get phones and other things from most of these services, some that can also act as WiFi hotspots for other devices- but we didn't really have any need for phone service so we just went with data only.
Quality of the services was really good, would have to be inside a big building or in the subway for service to get kind of spotty. Battery lasted about 8 hours straight, so usually had to charge up a little if using it all day long.
The only places we really had WiFi was in the hotels (which we were essentially paying for.) There were a lot of places that had WiFi, but required a subscription of some type. We would highly recommend this for anyone traveling to Japan, this link goes to the Narita airport site that lists some of the companies for these services- I'm sure they're all pretty good - Narita Airport cellphone/mobile wifi rentals