Cheap Switches, and Rebuilding Infrastructure for Reliability
Today we hastily gathered tools and volunteers and headed up to La Cresta to make some repairs and rebuild some facilities slightly ahead of schedule. This is a quick report on what happened and what we did today. Thanks again to all those that turned up to help on such short notice!
We received a report yesterday (Friday) of problems at La Cresta - stands to reason, since I'd just written how everything was stable there. Some phone debugging suggested that DHCP wasn't working.
To cut a long debug story short (including a site street-level visit by Drew and reset and reconfig of the m0n0wall relay/router), we decided to go onto the roof today.
Bottom line: a cheap switch we'd put in there after an emergency reconfig after the thunderstorms took out 2 802.11a backhauls, had decided to go weird. It was passing all traffic just fine, but not DHCP traffic.
It was really quite weird. All the lights were on the switch, except the lights where there were actual links present! Power cycling the switch fixed the problem. But what a pain to mount and climb a ladder to fix this!
Ironically, we recognised this after the thunderstorms, and after some consultation with the owner, had devised a solution. Instead of accessing everything from the roof, the owner installed a 20"×20"×6" (approx) box at ground level, with power in the box and conduit to the roof. And we ordered DSL for La Cresta so it would have its own feed and not have to rely upon a relay link.
All this was sitting there waiting for the DSL to start (next Friday), and cabling to be installed from the roof to ground level. So rather than simply go up to the roof and debug the dhcp problem (not knowing what it was) we decided to bite the bullet and install the new cabling and bring everything back to ground level, except what absolutely had to remain on the roof.
So most of the afternoon was spent pulling cable through conduit - after the obligatory run to Home Depot to get conduit fittings. We mostly got off the roof around dark, and finished crimping connectors and restarted everything around 6pm (about 3 hrs from switch off to switch on).
The end result is that there are no devices powered from the roof. We can power cycle everything from the ground. We can also plug into every cable on the ground - so debugging a DHCP problem next time would be a matter of going to the box with a laptop, and the ethernet switch (a new one!) could be reset right there.
- We have a big drive towards simplicity at the moment. Adding a switch increased the number of failure points. And adding a cheap switch that sat in a hot box, in the sun all day, made it worse.
- Physical Access is important to reliability. La Cresta has had more problems and longer downtimes than other nodes because it requires a ladder to reach it, and it was also behind a locked gate. This reduces the desire for ad-hoc fixes and cleanup, and increases the resources to get to it (someone with a key, 2 people to handle the ladder)
Our simplicity drive, falling DSL prices, and increased usage have all driven adding DSL feeds to more locations. By Friday, we'll have four DSL feeds for the ten buildings providing internet in Golden Hill. That's a big jump from the original one feed that at its peok was feeding 7 buildings!
Also driving this: two of the three landlords involved have stressed that they're now willing to pay more for reliability. Internet is less of a luxury and more of a necessity and they're willing to pay accordingly.
On their behalf, we're spending money on the new feeds, upgrading equipment to a common base, separating routers from radios (addming m1n1wall boxes) for simplicity in configuration and hopefully reliability, and upgrading physical infrastructure (e.g. getting a cage for the Pink Palace basement so we can store spares, common tools, as well as more readily access the installed equipment).
All of this means lots of volunteer opportunies. We'll try to announce them as far ahead of time as possible. Feel free to shoot me a line with your availability if you'd like me to email you directly.
Pics of the new work to follow later this week.