GIS, Macro-Thinking, Gaia-scale Biotech

[Justin Boland]

Admittedly cheeseball, but still the core statement: this is a struggle for Life -- biological, animal, vegetable, diverse flora and fauna Earth Life. We've come here from many paths, all of which involved community planning. I'm interested in Geographic Information Systems for community planning, but what's grabbing me this morning is the potential for global-scale strategic thinking.

It's not any stretch to say that some projects are more significant than others. There's hot-spots, turning points, critical bottlenecks...always exceptional circumstances. We could, however, streamline the process of identifying those circumstances using GIS.

Given the sheer variety of biomes, ecosystems and their interactions along boundary lines, we're looking nearly a thousand variations on the "Bioshelter Greenhouse" theme -- from the remarkable Seawater Greenhouse to the Earthship to the Living Machines that work nearly everywhere. Different natural resources call for different design approaches.

We already have the technology we need -- up next comes simplifying and improving communication, raising tech awareness / increasing "infrastructure literacy" and lowering entry costs.

[Justin Boland]

[Korey King]

this is ambitious as hell my friend. I can dig it.

so what you're looking for is something to pull data from around the world (climate/precip/insolation/wind, population, food prod, soil types, local avail. of materials, this could go on and on...) and visualize it for different projects and project needs? something like that?

sounds incredibly useful. looks like OSGeo/GeoTools is the dominant open-source GIS toolkit, can check 'em out here:
http://www.osgeo.org/geotools

gCensus might be a good framework to build upon/leech off/integrate with, free+works with google earth:
http://gecensus.stanford.edu/gcensus/index.html

and it looks like GE itself is easy enough to tack GIS data onto:
http://interactiveearth.blogspot.com/20 ... earth.html

[Justin Boland]

Yes, I'm curious how big some of these datasets are, and thus what the costs would be to store it all locally.

I am curious about how to combine multiple datasets, and it's a testament to my GIS format ignorance that I don't even know how to phrase that question. Can I just dump MapPoint, Google Earth, USGS and Weather Service data together on a bad-ass computer and start looking for connections?

I'm also digging back into "Radical Cartography," something I nibbled around back when I was doing skilluminati stuff all the time...gonna check out Subtopia once again.

This project area is something I was calling "Transitionware" back when I was sketching it out:

How much of reality are US citizens allowed to see? Can we make Google Earth into a three dimensional, searchable space that they can plug data sets into? Can this be done through an intuitive, beginner-level interface? Is it reasonable to expect all the information be recent -- not more than 3 months old, for instance?

Is it reasonable to expect that we all have access to the infrastructure of our communities? Do you trust yourself to walk along the shores of your local reservoir? Do you trust your kids with a computer-searchable, photo-accurate satellite map of your town?

How can government and corporate statistics be made more available and more visual? Is Adobe working on an application to gather and dramatically visualize data into quicktime movies and flash animation? Is somebody getting Christopher Alexander, Edward Tufte, Hans Rosling and William Gibson in the same room yet?

How much can joining be automated? Do we really need to fill out 10,000 profiles? Can we autojoin groups based on geotagged location? As the user, why should have have to start a profile, then go searching for local group? Obviously that's why the user is there. Make the obvious decisions automatic.

The first benefit of joining should be considerable: not "membership," not forum discussions, but a PDF report on their immediate ecosystem. Water quality reports, maps of water source proximity and purity, demographic reports and maps, detailed numbers on natural resources and economic history, lists of plants ideally suited to your climate, links to organizations already at work in your area with contact info.

I realize that last bit is a massive task but it's worth fucking with.

[Korey King]

http://data.geocomm.com/catalog/US/61082/group201.html

soil survey GIS data takes up between 5-40mb per county, call it an average of 20mb.
3,140 counties in the US, so about 63gigs for soil data alone.

hydrography takes a lot less (2-200kb statewide), it looks like most simple(r) datasets will be in the low KB range.
http://data.geocomm.com/catalog/US/61082/group105.html
http://data.geocomm.com/catalog/US/61082/datalist.html

actually getting the data and keeping it current may be fairly expensive: OSGeo's datasets can run about $500 for 8gb(!)
http://data.geocomm.com/premiumdownloads.html

there are other GIS data sources, all I've found so far do charge for some (if not all) of their data -- more intensive data like soil sampling and chronological stuff will need more storage, and will probably come with a fee.

geocomm has some free stuff: http://data.geocomm.com/catalog/index.html but the soil data was all premium.

I'm sure with enough digging you can find free data for just about anything though.
may have to resort to scraping and reformatting into a standard GIS format for use.

[Korey King]

I am curious about how to combine multiple datasets, and it's a testament to my GIS format ignorance that I don't even know how to phrase that question. Can I just dump MapPoint, Google Earth, USGS and Weather Service data together on a bad-ass computer and start looking for connections?

also have little info on the topic, but from my brief+basic understanding you'll probably need to reformat whatever data isn't a standard GIS format. if everything plays nice with whatever toolkit you use it should be as simple as layering each set. should be able to build it w/transparency if needed -- the data is just data, how you visualize it depends on how you build it. I think.

relevant parts of the massive wikipedia entry on GIS:

Data representation

GIS data represents real world objects (roads, land use, elevation) with digital data. Real world objects can be divided into two abstractions: discrete objects (a house) and continuous fields (rain fall amount or elevation). There are two broad methods used to store data in a GIS for both abstractions: Raster and Vector.

Raster

A raster data type is, in essence, any type of digital image represented in grids. Anyone who is familiar with digital photography will recognize the pixel as the smallest individual unit of an image. A combination of these pixels will create an image, distinct from the commonly used scalable vector graphics which are the basis of the vector model. While a digital image is concerned with the output as representation of reality, in a photograph or art transferred to computer, the raster data type will reflect an abstraction of reality. Aerial photos are one commonly used form of raster data, with only one purpose, to display a detailed image on a map or for the purposes of digitization. Other raster data sets will contain information regarding elevation, a DEM, or reflectance of a particular wavelength of light, LANDSAT.

Digital elevation model, map (image), and vector data

Raster data type consists of rows and columns of cells, with each cell storing a single value. Raster data can be images (raster images) with each pixel (or cell) containing a color value. Additional values recorded for each cell may be a discrete value, such as land use, a continuous value, such as temperature, or a null value if no data is available. While a raster cell stores a single value, it can be extended by using raster bands to represent RGB (red, green, blue) colors, colormaps (a mapping between a thematic code and RGB value), or an extended attribute table with one row for each unique cell value. The resolution of the raster data set is its cell width in ground units.

Raster data is stored in various formats; from a standard file-based structure of TIF, JPEG, etc. to binary large object (BLOB) data stored directly in a relational database management system (RDBMS) similar to other vector-based feature classes. Database storage, when properly indexed, typically allows for quicker retrieval of the raster data but can require storage of millions of significantly-sized records.

Vector
A simple vector map, using each of the vector elements: points for wells, lines for rivers, and a polygon for the lake.

In a GIS, geographical features are often expressed as vectors, by considering those features as geometrical shapes. Different geographical features are expressed by different types of geometry:

* Points

Zero-dimensional points are used for geographical features that can best be expressed by a single point reference; in other words, simple location. For example, the locations of wells, peak elevations, features of interest or trailheads. Points convey the least amount of information of these file types. Points can also be used to represent areas when displayed at a small scale. For example, cities on a map of the world would be represented by points rather than polygons. No measurements are possible with point features.

* Lines or polylines

One-dimensional lines or polylines are used for linear features such as rivers, roads, railroads, trails, and topographic lines. Again, as with point features, linear features displayed at a small scale will be represented as linear features rather than as a polygon. Line features can measure distance.

* Polygons

Two-dimensional polygons are used for geographical features that cover a particular area of the earth's surface. Such features may include lakes, park boundaries, buildings, city boundaries, or land uses. Polygons convey the most amount of information of the file types. Polygon features can measure perimeter and area.

Each of these geometries is linked to a row in a database that describes their attributes. For example, a database that describes lakes may contain a lake's depth, water quality, pollution level. This information can be used to make a map to describe a particular attribute of the dataset. For example, lakes could be coloured depending on level of pollution. Different geometries can also be compared. For example, the GIS could be used to identify all wells (point geometry) that are within 1-mile (1.6 km) of a lake (polygon geometry) that has a high level of pollution.

Vector features can be made to respect spatial integrity through the application of topology rules such as 'polygons must not overlap'. Vector data can also be used to represent continuously varying phenomena. Contour lines and triangulated irregular networks (TIN) are used to represent elevation or other continuously changing values. TINs record values at point locations, which are connected by lines to form an irregular mesh of triangles. The face of the triangles represent the terrain surface.

Advantages and disadvantages

There are advantages and disadvantages to using a raster or vector data model to represent reality. Raster datasets record a value for all points in the area covered which may require more storage space than representing data in a vector format that can store data only where needed. Raster data also allows easy implementation of overlay operations, which are more difficult with vector data. Vector data can be displayed as vector graphics used on traditional maps, whereas raster data will appear as an image that, depending on the resolution of the raster file, may have a blocky appearance for object boundaries. Vector data can be easier to register, scale, and re-project. This can simplify combining vector layers from different sources. Vector data is more compatible with relational database environments. They can be part of a relational table as a normal column and processed using a multitude of operators

The file size for vector data is usually much smaller for storage and sharing than raster data. Image or raster data can be 10 to 100 times larger than vector data depending on the resolution. Another advantage of vector data is that it is easy to update and maintain. For example, a new highway is added. The raster image will have to be completely reproduced, but the vector data, "roads," can be easily updated by adding the missing road segment. In addition, vector data allows much more analysis capability, especially for "networks" such as roads, power, rail, telecommunications, etc. For example, with vector data attributed with the characteristics of roads, ports, and airfields, allows the analyst to query for the best route or method of transportation. In the vector data, the analyst can query the data for the largest port with an airfield within 60 miles and a connecting road that is at least two lane highway. Raster data will not have all the characteristics of the features it displays.

Non-spatial data

Additional non-spatial data can also be stored along with the spatial data represented by the coordinates of a vector geometry or the position of a raster cell. In vector data, the additional data contains attributes of the feature. For example, a forest inventory polygon may also have an identifier value and information about tree species. In raster data the cell value can store attribute information, but it can also be used as an identifier that can relate to records in another table.

There is also software being developed to support spatial and non-spatial decision-making. In this software, the solutions to spatial problems are integrated with solutions to non-spatial problems. The end result it is hoped with these Flexible Spatial Decision-Making Support Systems (FSDSS)[9] will be that non experts can use GIS and spatial criteria with their other non spatial criteria to view solutions to multi-criteria problems that will support decision making.

[Korey King]

Open Access to USGS DEM data... Myth Or Reality? You Be The Judge
http://www.gisuser.com/content/view/7797/28/

I'm very new at mapping so excuse my ignorance if I'm totally off base.
I read this about getting SDTS DEM files on the USGS webpage:

1) The original DEM 7.5 minute tiled data available only in Spatial Data Transfer Standard (

And the reasoning for this is:

The decision to provide SDTS DEMs through the GIS Data Depot is consistent with the USGS goal of making more public domain data available at no charge while enabling the private sector to have a greater role in the distribution of USGS data. SDTS DEMs placed on media in response to customer orders will be sold by the GeoCommunity at a price less than the USGS currently charges.

Looking at datadepot, it costs $995 to buy SDTS DEM on CD for the entire USA. You can download it for free but you have to click a million links to do it and they don't allow web spiders to get the data automatically. It'll take months to manually download it all.

Looking at mapmart I can't see any clear way to get it for free.

And looking at ATDI-us you have to select a lat/long, again making it difficult to get the entire data set.

Can we just have a FTP server where we can get the entire lot? These private sector implementations are either very poorly implemented or purposely designed to make it very difficult to get the entire data set.

A USGS representative graciously took some time to provide a detailed response. It read as follows:

We greatly appreciate the comments and suggestions from the members of the list. Many of you have been around longer than me, and I've been a part of the list for about six years now. Your participation and suggestions have been a large part of why the SDTS DEM dataset remains an important and viable dataset. Thank you.

Although going through our commercial venders for SDTS DEMs may not always be the easiest or most efficient method, they do provide the data free of charge. It would have been great if the USGS was able to continue serving these data via USGS ftp site, but unfortunately we were not. Please keep in mind that commercial vendors are in business to make a profit. I, for one, greatly appreciate the time and effort put forth by our three venders, MapMart, GeoCommunity, and ATDI, to continue to offer USGS SDTS DEMs free of charge.

I'm not sure what the going rate is to host and maintain a server with several hundred gigabytes of data, and handling hundreds of thousands of file downloads in addition to providing monthly maintenance, but it must be substantial. [actually, the cost for such bandwidth coupled with applications developed using Open Source solutions are substantially less than now than when these arrangements were first setup so it really isn't that cost prohibitive?]

We welcome other entities to partner with the USGS in serving SDTS DEMs free of charge to the public, but for now, MapMart, GeoCommunity, and ATDI have been the only three to enter into an agreement with the USGS.

consider the Florida GIS Data Sharing Network (see http://groups.google.com/group/GIS-Data). This popular group brings together GIS professionals willing to share their data... what great idea! Think about it... you download or buy some DRGs, DEMs, DOQs, parcel data, or what have you. USing the list you can easily arrange to swap, share, exchange, or give away your GIS data needed by other professionals for their projects. To date the project appears to have been quite succesful and I'm actually quite surprised that other such cooperatives haven't popped up (if so, please tell me about it). For other data sharing cooperatives see http://www.gis-data.blogspot.com/ & a good example from Palm Beach County.

1:24,000 DEMs are easily available from the likes of the New Jersey State GIS Clearinghouse. These data are conveniently listed by USGS Quad name and as an added bonus, official FGDC metadata is also provided. Similarly, pretty much every other State clearinghouse has these data available, typically via FTP download... aka. grab what you need... woohoo! Look around and you'll likely find the data in .DEM format rather than the pesky native SDTS format

Users should also consult the Seamless data distribution system at http://seamless.usgs.gov where you can access the National Elevation Dataset (NED), in particular, the 1/3 arcscond data coverage. NED 1/3 arc second (~ 10 meter resolution) is processed and assembled the same as the NED 1 arc second, but the source data is 10 meter elevation data or higher resolution. Of interest, The coverage area is approximately 70% of the United States. There are 1/9 arcsecond data, however, the coverage is minimal. NED 1/9 arc second (~ 3 meter resolution)is processed and assembled the same as the NED 1 and 1/3 arc second, but the source data is using mostly LIDAR elevation data sources. Of note, the NED is derived from USGS DEM data.

Finally, another source that you absolutely MUST consult is the USDA Geospatial Gateway - home to NAIP ortho photography, DEMs, and a "whackin pant-load" of other spatial datasets - See http://datagateway.nrcs.usda.gov/GatewayHome.html

be sure to sink your teeth into http://gisuser.com and chew awhile -- awesome resource.

http://www.gisuser.com/content/view/607/28/-- The GISuser's Guide to locating and downloading Free USGS data
http://www.gisuser.com/content/view/2379/28/ -- State GIS Clearinghouse Directory
http://gisdata.blogspot.com/ -- Public domain GIS data and Free GIS data repositories and clearinghouses.

[Korey King]

ok. becoming pretty clear that the data is out there and available (at least for the US) -- might be a good idea to think about what data is needed here to hunt down whoever's got what and get a more solid figure on storage/server costs. everything's a good answer, but let's start with the essentials:

Geological
climate, elevation, soil, rainfall, wind/solar/geothermal potential, mineral/metal/natural resources, native foods+wildlife, pollutants+locations, ...

Financial
imports/exports/manufacturing/food production locations, unoccupied/foreclosed/dilapidated property, ...

Populitical
census, employment, roads+bridges+related infrastructure, health+disease+treatment trends, ...

also becoming clear that there will not be standard GIS datasets for a lot of essential stuff, so rolling your own will be necessary.
well. is that the point?

not just streamlining the accessibility of all this data useful to developers/rehabilitators, but making it simple to add/update/collaborate new data?

[Justin Boland]

The goal I have in mind is public intelligence as a non-profit service, definitely. People deserve to get this information quickly and for free -- at least to me, that seems both useful and feasible enough to be a worthy project. Especially since the tech requirements are lower than I expected. We can build a monster system for less than $10k.

24" Monitor - $375 ( $359.99 + shipping guesstimate)
http://www.jr.com/hp/pe/HP_W2408H/

or a 25.5" Sony Vaio system with 8 GB ram: $3700
http://www.jr.com/sony/pe/SON_VGCRT150Y/

5 of these LaCie 5 TB Cubes: $5,250 or so
http://www.jr.com/lacie/pe/LAC_301355U/

[Justin Boland]

Or, holy f'ing s*** thank you Zach, we could daisy chain Playstations together:

http://www.wired.com/techbiz/it/news/20 ... ercomputer

It turns out that the PS3 is ideal for doing precisely the kind of heavy computational lifting Khanna requires for his project, and the fact that it's a relatively open platform makes programming scientific applications feasible.

"The interest in the PS3 really was for two main reasons," explains Khanna, an assistant professor at the University of Massachusetts, Dartmouth who specializes in computational astrophysics. "One of those is that Sony did this remarkable thing of making the PS3 an open platform, so you can in fact run Linux on it and it doesn't control what you do."

He also says that the console's Cell processor, co-developed by Sony, IBM and Toshiba, can deliver massive amounts of power, comparable even to that of a supercomputer -- if you know how to optimize code and have a few extra consoles lying around that you can string together.

"The PS3/Linux combination offers a very attractive cost-performance solution whether the PS3s are distributed (like Sony and Stanford's Folding@home initiative) or clustered together (like Khanna's), says Sony's senior development manager of research and development, Noam Rimon.

http://www.physorg.com/news148749271.html

And the Motherlode itself:
http://www.ps3cluster.org/index.html

Our community guide allows you to set up your own MPI (Message Passing Interface) based supercomputer cluster with the Playstation 3. This guide was co-written by Gaurav Khanna, based on his previous work on the Gravity Grid and is a current run-time environment for the research of co-author (Chris Poulin), based on his current work in distributed pattern recognition. As such, we currently utilize the Fedora Core for this infrastructure and illustrate a "how-to" below. NOTE: We focus on the Fedora 8 distribution, due to prevalence of Fedora and its Cell SDK (3.0) compatibility. Finally, this content should be considered open source, and here is the license.

[Korey King]

distributed computing w/playstations would be utterly awesome -- especially if there's a way to utilize them as is (without modding/installing an OS). I've run folding@home for a few years now and there is almost no perceptible performance loss, but even if they're only utilized when idle it would still be great to use for as-needed CPU cycles and even better -- massively redundant storage.

whether or not the data processing needed for this project is viable for distribution is def. something to look into -- but even if the load isn't distributed PS3s are a'plenty and will become obsolete pretty quick -- would be a great (tax-deductible) incentive for people to donate them as they become obsolete, allowing the system to scale as more data is added.

is there a pet name for this project floating around yet?

[Justin Boland]

Currently nameless, although I have a buddy advocating for Geographic Information System Synergy-Unity Machine, which leads to an unfortunate acronym.

[Justin Boland]

http://www.worldmapper.org/data.html

^^World Mapper's database. I definitely think that's a half-baked project -- the distortions of their viz. system make it just worthless to me, nothing is recognizable enough and the color palette is pretty wack. Yeah, I said it. There's much better uses for that data, I'm just not qualified to pull them off. Yet. Gonna pass this on to a few dudes I know who possibly could.

Also, pics on food imports. Been thinking about Libya -- forward-thinking Oasis in the middle of Africa's worst food problems. I'm interested in checking out the permaculture horizons there. It would be awesome to get some land there and work on extreme conditions -- integrated moisture collection and recycling, indoor aquaponic systems, and intensive composting all come to mind.

[Adrisya Alok]

I am positive this will be interesting to the both of you -- "Knowledge Mapping for Open Sense-making Communities"
http://www.slideshare.net/sbs/knowledge ... ommunities

Great slide presentation on the power of maps and re-thinking how to organize and teach information.

[Adrisya Alok]

Also, since activity has slowed on this thread, allow me to ask: are you still working on plans for an actual GIS workstation/lab? I have several friends and contacts who work in the field and I could ask them for equipment lists and such. If either of you are already working on that, I shall avoid being redundant. Let me know if you'd be interested.

[Korey King]

absolutely, it would be great to see what they work with. I haven't dug very deep into workable actuals for GISSUM (ha) yet, but justin's all over the place and I wouldn't be surprised if he's got a lot planned -- but that's no reason not to be redundant. but.

[Justin Boland]

Korey is definitely right: for all aspects of this project, redundancy is an asset, not a waste.

If we're all typing up our own manifestos to distribute? That's awesome, then we've got more data to synthesize with...more examples to learn from, more testing to dowse for #whatworks.

Got a reply from the local University lab:

Thanks for the inquiry. Our lab has two scanners. One a large flat scanner and the other a slide scanner. They are both manufactured by Microtek. We also have a server and three computer stations in the lab. Additionally, we have access to a classroom with approximately 30 terminals where we hold workshops and courses.

A plotter printer is vital for map printing. We have a 42 HP printer. Also, we have assorted Magellan GPS handhelds and one very nice Trimble GPS unit. As far as the software at the very least, you would need a high-end GIS program. We use ESRI v.9.2. I would also suggest Photoshop and the usual MS Office Suite to include Access and Excel. This should give you an idea of what you might need to get started. I hope this helps and please feel free to contact me if you have any other questions. In addition, you might try contacting UIUC or SIU and inquire as to their equipment/software. Thanks.

[Justin Boland]

Source:
http://housewithoutwalls.wordpress.com/ ... coalition/

im not sure, but something of that sort would be really sweet. there are a lot of GIS tools out there. ArcGIS is a good place to start, and with enough practice you can get pretty efficient with it. I’m sure that there are tie-ins like that oriented toward permaculture and urban gardening and green geurillas kinda stuff. As a campus landscaper I was always proud to leave food plants hidden by folks when weeding. GIS would make those kind of operations a lot easier, especially when dealing with parks, rooftops, hidden nooks and crannies, etc. Its also great for planning urban and peri-urban foodsheds. I’m not super familiar with GIS but im sure many down geographers and ecologists/ environmental scientists about your town have some competence with the software.

As far as visualizing software in general goes, however, I would totally recomend Compendium for knowledge and concept mapping. Check it out and EcoSensus, where they integrate Udig, some software i dont yet understand.

I would totally agree that visualization is key and it will allow for people to better communicate across bioregions, manage resources, make decisions and plan and organize. The key is breaking with linear narratives and organizing information, ideas, arguments and concepts in ways that make more sense to us.