The deep or hidden web can be defined in the following way: web content that is not part of the surface web which is indexed using standard search engines (see: Exploring the Deep or Hidden Web with WebMynd). Exposing and utilizing hidden web content has been a goal of computer scientists and entrepreneurs since web search using Google and other search engines became commonplace. A recent article explains a new facet of this quest (see: This Data Isn’t Dull. It Improves Lives). Below is an excerpt from it regarding previously hidden "governmental" information:
Take data that you and I have already paid a government agency to collect, and post it online in a way that computer programmers can easily use. Then wait a few months. Voilà! The private sector gets busy, creating Web sites and smartphone apps that reformat the information in ways that are helpful to consumers, workers and companies. Not surprisingly, San Francisco, with its proximity to Silicon Valley, has been a pioneer in these efforts. For some years, Bay Area transit systems had been tracking the locations of their trains and buses via onboard GPS. Then someone got the bright idea to post that information in real time. Thus the delightful app Routesy was born. Install it on a smartphone and the app can tell you that your bus is stuck in traffic and will be 10 minutes late ....It gives consumers a great new way to find out when and where the bus is coming, and all at minimal government expense. Another example involves weather data produced by the National Oceanographic and Atmospheric Administration. The forecasts you find on the Weather Channel, or on the evening news or online, use the agency’s information. Again, the government produces and releases raw data, and the private sector transforms it into something useful for the public. Several other departments in the Obama administration are looking to expand the use of such techniques. On data.gov, you will find huge amounts of downloadable data that had heretofore been inaccessible....Now the administration is pushing to use this concept as a tool for regulation, and as a method of avoiding more heavy-handed rule making. The idea is that making things more transparent can immediately turn consumers into better shoppers and make markets work better.
The lesson to be learned here is straightforward -- make data collected by local, state, and regional governmental bodies accessible and companies in the private sector will find ways to make it useful to the general public, particularly using smartphone and tablet apps. I am sure that there will be naysayers to this proposition in the public sector, saying, for example, that companies should not derive profit from such public data. This, of course is not a valid argument because value is being added and the general public can derive benefit from the previously hidden, and therefore unusable, data. This is all about accessibility and transparency. In fact, this very transparency that may be the basis for opposition to accessing the hidden web data.
Comments