Use the Ned idea of restricting access by IP ranges to ease the overload peaks. A bit like a rolling blackout!
Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?
I prefer the last option as Boinc is supposed to be unobtrusive by design.
A combination of all the above options would be the most thorough traffic solution.
And again... Is not the grey fibre already inplace? It's just a question of replacing the equipment at both ends on campus.
Isn't the biggest headache the db bottleneck and maintenance nightmare that appears to be quickly heading towards a 24hour operation in itself.
Do we get a duplicate Matt when the maintenance burden exceeds 24hours in any 24hour period?
Message 916134 - Posted: 9 Jul 2009, 12:28:30 UTC - in response to Message 916120.
Boinc has two different methods - the exponential backoff and the project requested delay.
If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.
Message 916136 - Posted: 9 Jul 2009, 12:48:51 UTC - in response to Message 916134.
That's only one aspect and works with a similar effect as does the Ned rolling IP 'blackout' idea to reduce the number of simultaneous requests.
What I'm proposing is traffic management of the data rates so that whatever network bottleneck is suffered wherever, the clients reduce their data rates for themselves to a low enough level such that they themselves no longer see packet loss or resend requests at the TCP level. In other words, implement traffic shaping from the client side to avoid a link overload. Hit a backoff timeout if the data rate has to be tuned down to an unrealistically long send time.
In this way, Boinc can learn for itself how to best utilise limited network bandwidth as demand varies, and all done automatically with no human 'guesswork' required.
A big problem is that the effective bandwidth of an overloaded link is very much less than the physical link bandwidth due to all the wasteful resends required when even just a small few data packets get dropped/lost due to overload.
Message 916145 - Posted: 9 Jul 2009, 13:48:31 UTC - in response to Message 915800.
would be eligible to apply for some of this NSF Money ?
to run a Fiber Optic Cable up the hill to the Space Sciences Laboratory (SSL) at Berkeley ?
Inter-Campus and Intra-Campus Cyber Connectivity ?
The Experimental Program to Stimulate Competitive Research (EPSCoR) is a program designed to fulfill the National Science Foundation's (NSF) mandate to promote scientific progress nationwide. The EPSCoR program is directed at those jurisdictions that have historically received lesser amounts of NSF Research and Development (R&D) funding. Twenty-seven states, the Commonwealth of Puerto Rico and the U. S. Virgin Islands are currently eligible to participate. Through this program, NSF establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state's or region's research infrastructure, R&D capacity and hence, its national R&D competitiveness.
The American Recovery and Reinvestment Act of 2009 will enable NSF to invest $20 million in the Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2). Awards made under this program will provide up to $1 million for up to 2 years to support the enhancement of inter-campus and intra-campus cyber connectivity within an EPSCoR jurisdiction. These awards are intended to enhance broadband access for academic research and the utilization of cyberinfrastructure consistent with the jurisdiction's Science and Technology (S&T) plan. The inter-campus and intra-campus connectivity targeted by these awards is expected to broaden individual and institutional participation in STEM research and education activities within and among jurisdictions and to facilitate synergy among NSF EPSCoR Research Infrastructure Improvement activities.
Message 916164 - Posted: 9 Jul 2009, 14:46:00 UTC - in response to Message 916153.
Everyone keeps forgetting the ability of Boinc to Cache work. As long as user are able to download up to 10 days of work, it's possible ride out any outage that is less than 10 days. I suspect a good solution would be to limit the Cache to one or two days so the data base clutter caused by unprocessed work units is limited but users can survive the several hour long outages that we often see. As for the link speed, not to long ago we were living with a 10 meg link. This is a problem that can be solved by throwing money at it. Would you donate $10 to fix the problem? Thats all it would take if enough people donate. With our numbers, we don't need large donations, we just need many small ones.
If the big boys at UC Berkley say, Ok seti you can have your Gig pipline but you need to pay for it i will more than gladly donate. And it would be a tad more than $10.00.
the fact is, as this or any program grows, it eventually needs a bandwidth and server upgrades. look at how many times the hotmail servers and network links have been upgraded over the years for an example.
seti will never be at a point where you can say there is enough bandwith. as more and more people come online, the pipe slowly gets clogged up again. even if we has a gigabit link to the outside world, this would only be good for a few years. imagine how fast computers will be a few years from now. what currently takes a cuda card 6 minutes might take 6 seconds.
so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.
Message 916187 - Posted: 9 Jul 2009, 16:02:15 UTC - in response to Message 916183.
So when they put the fiber in, it should be the fastest fiber they can afford.
Message 916206 - Posted: 9 Jul 2009, 17:06:58 UTC - in response to Message 916187.
i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.
i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.
basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.
i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?
i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.
i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.
i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).
Message 916246 - Posted: 9 Jul 2009, 19:33:53 UTC - in response to Message 916109.
Message 916248 - Posted: 9 Jul 2009, 19:37:21 UTC - in response to Message 916187.
As a general rule, the cost of "pulling fiber" is mostly labor.
So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.
Message 916259 - Posted: 9 Jul 2009, 20:25:57 UTC - in response to Message 916248.
And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.
Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.
Message 916287 - Posted: 9 Jul 2009, 20:59:25 UTC - in response to Message 916259.
And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.
Message 916290 - Posted: 9 Jul 2009, 21:01:00 UTC - in response to Message 916206.
i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do.
All of this is observation, or based on my recollection of statements made by SETI@Home staff. Errors are mine, and my thoughts don't represent U.C. Berkeley or SETI@Home.
If you do a traceroute to setiathome.ssl.berkeley.edu and another traceroute to setiboincdata.ssl.berkeley.edu, you'll find very different paths.
The forums run using "campus" bandwidth, so the forums and the statistics links and etc. don't touch the 100 megabit bandwidth.
What they do use is database access, and the database is one of the bottlenecks.
SETI@Home buys bandwidth "off campus" because of what Campus charges them for "campus" bandwidth. The bandwidth through Hurricane Electric is much cheaper, and Campus has been gracious enough to route the off-campus bandwidth to the lab. Here is an article from 2002: http://www.mail-archive.com/setiathome@klx.com/msg01049.html.
U.C. Berkeley owns their own facilities. To bring in outside "wire" is going to cost a lot more than extending campus facilities.
The lab was built in 1966, and I'm sure they pulled in plenty of phone lines based on 1966 standards. Today, they likely would not pull in bundles and bundles of copper, but would pull a healthy bundle of fiber.
Message 916291 - Posted: 9 Jul 2009, 21:02:27 UTC - in response to Message 916206.
Let me clear something up. The website is setiathome.berkeley.edu and is on the normal campus bandwidth. The upload/downloads are on ssl.berkeley.edu and are on the Huricane Electric bandwidth SETI pays for. Shutting down the forums doesn't get any additional bandwidth.
Now you ask why were they down? Load on the database servers. All those links we have to our work units and the stats are a load on the database server. They are the reason that sometimes it makes sense after the weekly outage to leave them off until the replica DB catches itself back up to sync. The stats are normally served off the replica and not the master.
Message 916319 - Posted: 9 Jul 2009, 21:51:12 UTC - in response to Message 916287.
...Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.
Using the existing grey fibre looks to be the best bet.
With DWDM, you can get 8 Tb/s down one fibre... Could the Berkeley servers keep up with that?
... OK, so perhaps 192 Tb/s isn't actually needed right now, but there are 1.5 Gb/s fibre modems readily available.
Message 916322 - Posted: 9 Jul 2009, 21:55:51 UTC - in response to Message 916319.
Currently, the Southern sky is very under-represented in the SETI data. Would there be any benefit in encouraging the Australians (for instance) to bid for money to set up a Southern SETI to distribute and analyse data from the Parkes radio telescope? Thus, if clients were unable to connect to Berkeley, they wouldn't necessarily sit idle, because they'd then automatically check if Parkes had any WUs to crunch.
Obviously, this wouldn't happen overnight because it depends upon funding and peer review. Nevertheless, increasing the amount of sky surveyed and (importantly) increasing the number of distribution hubs for SETI WUs would reduce the pressure on the Berkeley servers.
On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data? Perhaps a model of multiple distribution servers, but one receiving server might simplify things?
Of course, everything depends on funding; but if no-one tries submitting the grant applications then nothing happens.