From mpalmer at hezmatt.org Thu Sep 1 01:36:57 2016 From: mpalmer at hezmatt.org (Matt Palmer) Date: Thu, 1 Sep 2016 11:36:57 +1000 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: References: Message-ID: <20160901013657.GE4869@hezmatt.org> On Wed, Aug 31, 2016 at 10:45:48AM -0800, Royce Williams wrote: > Hypothetically, it would be an interesting strategy for a CA to > publicly demonstrate this level of competence: > > https://www.schrauger.com/the-story-of-how-wosign-gave-me-an-ssl-certificate-for-github-com > > ... while at the same time taking over another large install base like > StartSSL's (an install base fueled by offering free certs). > > If one got caught doing something naughty, one could buy time by A) > playing the incompetence card a few times, and B) having a large > enough deployment that it becomes non-trivial for the browsers/OSes to > revoke you outright. Honest Achmed's business model wins again! I'm pretty sure that's how this is going to go down here, too, incidentally -- there's just waaaay too many sites using WoSign (and StartCom) for the CAs' roots to just be pulled. Sad, but true. > Also, this is a cautionary tale about certificate diversity. > > Because of relative issuer stability, orgs have had the luxury of > depending wholly on a single cert supplier. The risk/continuity folks > might want to model some "one of our major certificate issuers just > got globally revoked" scenarios - if they haven't already. I'd be surprised if most business continuity people could even name their cert provider, and most probably don't even know how certs come to exist or that they *can* be made useless on a wide scale by the actions of, seemingly, an unrelated third party. It's a system nearly without precedent, when you think about it. In fact, my gut feel is that, if they really understood the system, most risk/continuity folks would scream "are you f**king kidding me? That's ridiculous!". Thanks, Netscape. Great ecosystem you built. - Matt -- Talk about unlucky. D'you know, if I fell in a barrel of tits I'd come out sucking me thumb. -- Seen on the 'net: http://thelawwestofealingbroadway.blogspot.com/2006/01/bang-to-rights.html From eric.kuhnke at gmail.com Thu Sep 1 01:42:48 2016 From: eric.kuhnke at gmail.com (Eric Kuhnke) Date: Wed, 31 Aug 2016 18:42:48 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901013657.GE4869@hezmatt.org> References: <20160901013657.GE4869@hezmatt.org> Message-ID: "Too big to fail" Where have we heard that before? If business risk/continuity people knew not only how much of a single point of failure a root CA is, but other basic stuff like "Maybe it shouldn't be possible to login to your domain registrar's control panel with the password known by Bob from Accounting, who wrote his pet's name down on a post-it note that he keeps in his desk drawer, and then point all the NS1/NS2/NS3 and glue records somewhere else..." On Wed, Aug 31, 2016 at 6:36 PM, Matt Palmer wrote: > On Wed, Aug 31, 2016 at 10:45:48AM -0800, Royce Williams wrote: > > Hypothetically, it would be an interesting strategy for a CA to > > publicly demonstrate this level of competence: > > > > https://www.schrauger.com/the-story-of-how-wosign-gave-me- > an-ssl-certificate-for-github-com > > > > ... while at the same time taking over another large install base like > > StartSSL's (an install base fueled by offering free certs). > > > > If one got caught doing something naughty, one could buy time by A) > > playing the incompetence card a few times, and B) having a large > > enough deployment that it becomes non-trivial for the browsers/OSes to > > revoke you outright. > > Honest Achmed's business model wins again! > > I'm pretty sure that's how this is going to go down here, too, incidentally > -- there's just waaaay too many sites using WoSign (and StartCom) for the > CAs' roots to just be pulled. Sad, but true. > > > Also, this is a cautionary tale about certificate diversity. > > > > Because of relative issuer stability, orgs have had the luxury of > > depending wholly on a single cert supplier. The risk/continuity folks > > might want to model some "one of our major certificate issuers just > > got globally revoked" scenarios - if they haven't already. > > I'd be surprised if most business continuity people could even name their > cert provider, and most probably don't even know how certs come to exist or > that they *can* be made useless on a wide scale by the actions of, > seemingly, an unrelated third party. It's a system nearly without > precedent, when you think about it. In fact, my gut feel is that, if they > really understood the system, most risk/continuity folks would scream "are > you f**king kidding me? That's ridiculous!". > > Thanks, Netscape. Great ecosystem you built. > > - Matt > > > -- > Talk about unlucky. D'you know, if I fell in a barrel of tits I'd come out > sucking me thumb. > -- Seen on the 'net: > http://thelawwestofealingbroadway.blogspot.com/2006/01/bang-to- > rights.html > > From lyndon at orthanc.ca Thu Sep 1 01:49:17 2016 From: lyndon at orthanc.ca (Lyndon Nerenberg) Date: Wed, 31 Aug 2016 18:49:17 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901013657.GE4869@hezmatt.org> References: <20160901013657.GE4869@hezmatt.org> Message-ID: > On Aug 31, 2016, at 6:36 PM, Matt Palmer wrote: > > Thanks, Netscape. Great ecosystem you built. Nobody at that time had a clue how this environment was going to scale, let alone what the wide-ranging security issues would be. And where were you back then, not saving us from our erroneous path ... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From marka at isc.org Thu Sep 1 03:06:03 2016 From: marka at isc.org (Mark Andrews) Date: Thu, 01 Sep 2016 13:06:03 +1000 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: Your message of "Wed, 31 Aug 2016 18:49:17 -0700." References: <20160901013657.GE4869@hezmatt.org> Message-ID: <20160901030603.E87CC530CCF9@rock.dv.isc.org> In message , Lyndon Nerenberg writes: > > On Aug 31, 2016, at 6:36 PM, Matt Palmer wrote: > > > > Thanks, Netscape. Great ecosystem you built. > > Nobody at that time had a clue how this environment was going to scale, > let alone what the wide-ranging security issues would be. > > And where were you back then, not saving us from our erroneous path ... Well lots of people have been pointing out the risks for years. We are no where at "to big to fail" here. We also have TLSA which can be used to prevent spoofed CERTs being successful. If you have a CERT you should be publishing a TLSA records and have it DNSSEC signed. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From george.herbert at gmail.com Thu Sep 1 04:33:18 2016 From: george.herbert at gmail.com (George William Herbert) Date: Wed, 31 Aug 2016 21:33:18 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901013657.GE4869@hezmatt.org> References: <20160901013657.GE4869@hezmatt.org> Message-ID: > On Aug 31, 2016, at 6:36 PM, Matt Palmer wrote: > > there's just waaaay too many sites using WoSign (and StartCom) for the > CAs' roots to just be pulled. Sad, but true. Not even. Pull away. > I'd be surprised if most business continuity people could even name their > cert provider, and most probably don't even know how certs come to exist or > that they *can* be made useless on a wide scale by the actions of, > seemingly, an unrelated third party. Not in my neck of the woods. If you have a drought of good ones in your area my consulting company calls that an opportunity... Sent from my iPhone From owen at delong.com Thu Sep 1 06:26:39 2016 From: owen at delong.com (Owen DeLong) Date: Wed, 31 Aug 2016 23:26:39 -0700 Subject: IPv4 Broker In-Reply-To: <26D0E6B0-5A17-47E2-AFAB-544E3983334A@gmail.com> References: <26D0E6B0-5A17-47E2-AFAB-544E3983334A@gmail.com> Message-ID: > On Aug 31, 2016, at 14:46 , Julien wrote: > > Between this two region it should be possible. > For sure, you can?t with a RIPE block as it?s not allowed to transfer from / to RIPE. Since when? Last I looked, ARIN was processing NRPM section 8.4 transfers with both RIPE and APNIC. Neither LACNIC nor AfriNIC have yet developed a compatible needs based transfer policy. Owen > > Julien. > >> On Aug 31, 2016, at 4:43 PM, Tyler Conrad wrote: >> >> Is there a way to fix the geolocation on a JPNIC netblock being advertised in the US market short of transferring the ownership of the record to an ARIN ASN? >> >> On Wed, Aug 31, 2016 at 2:37 PM, Julien > wrote: >> Hi, >> >> Just worked with couple of those in the RIPE list. >> Giving you the name off list might be better. >> >> If you don?t have yet anything from RIPE or APNIC, you can get a /22. >> You?ll have, at least, to subscribe as an LIR and pay the fees. >> It will be much less expensive than buying with a broker. >> You?ll still have the Geo Localisation issue if you want to use it in US. >> >> Julien. >> >> >> >>> On Aug 31, 2016, at 4:26 PM, Martin Hannigan > wrote: >>> >>> Hi Lorenzo, >>> >>> You can obtain a last /22 from the RIPE region as long as you comply >>> with their policy, yes, even if you are "in" the United States. >>> >>> http://bit.ly/RIPE22-20160831 >>> >>> That reduces your need and spend, unless you already received it. >>> >>> Then use the link Marco posted. It's a good list. >>> >>> And reference the archives as Cameron noted. >>> >>> >>> Best Regards, >>> >>> -M< >>> >>> >>> >>> On Tue, Aug 30, 2016 at 11:43 AM, Lorenzo Mainardi >>> > wrote: >>>> Do you know any good IPv4 broker? >>>> I need a /20. >>>> Regards >>>> >>>> >>>> >> >> > From mpalmer at hezmatt.org Thu Sep 1 10:10:17 2016 From: mpalmer at hezmatt.org (Matt Palmer) Date: Thu, 1 Sep 2016 20:10:17 +1000 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: References: <20160901013657.GE4869@hezmatt.org> Message-ID: <20160901101017.GW4869@hezmatt.org> On Wed, Aug 31, 2016 at 09:33:18PM -0700, George William Herbert wrote: > > On Aug 31, 2016, at 6:36 PM, Matt Palmer wrote: > > there's just waaaay too many sites using WoSign (and StartCom) for the > > CAs' roots to just be pulled. Sad, but true. > > Not even. Pull away. Not going to happen. Feel free to argue otherwise in the appropriate venues, but you're tilting at windmills, IMO. > > I'd be surprised if most business continuity people could even name their > > cert provider, and most probably don't even know how certs come to exist or > > that they *can* be made useless on a wide scale by the actions of, > > seemingly, an unrelated third party. > > Not in my neck of the woods. If you have a drought of good ones in your > area my consulting company calls that an opportunity... How the hell do you get from "the world does not work that way" to "please pitch me your consulting services"? - Matt From mpalmer at hezmatt.org Thu Sep 1 10:12:07 2016 From: mpalmer at hezmatt.org (Matt Palmer) Date: Thu, 1 Sep 2016 20:12:07 +1000 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: References: <20160901013657.GE4869@hezmatt.org> Message-ID: <20160901101207.GX4869@hezmatt.org> On Wed, Aug 31, 2016 at 06:49:17PM -0700, Lyndon Nerenberg wrote: > > On Aug 31, 2016, at 6:36 PM, Matt Palmer wrote: > > > > Thanks, Netscape. Great ecosystem you built. > > Nobody at that time had a clue how this environment was going to scale, > let alone what the wide-ranging security issues would be. Nor did they particularly trouble themselves to find out. > And where were you back then, not saving us from our erroneous path ... You're going to go with that one, are you? Good for you. - Matt -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 811 bytes Desc: Digital signature URL: From bortzmeyer at nic.fr Thu Sep 1 10:19:51 2016 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Thu, 1 Sep 2016 12:19:51 +0200 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901013657.GE4869@hezmatt.org> References: <20160901013657.GE4869@hezmatt.org> Message-ID: <20160901101951.usbucglng5itramz@nic.fr> On Thu, Sep 01, 2016 at 11:36:57AM +1000, Matt Palmer wrote a message of 45 lines which said: > I'd be surprised if most business continuity people could even name > their cert provider, And they're right because it would be a useless information: without DANE, *any* CA can issue a certificate for *your* domain, whether you are a client or not. From dot at dotat.at Thu Sep 1 13:00:25 2016 From: dot at dotat.at (Tony Finch) Date: Thu, 1 Sep 2016 14:00:25 +0100 Subject: Don't press the big red buttom on the wall! In-Reply-To: <20160830142110.GA792@sizone.org> References: <20160830142110.GA792@sizone.org> Message-ID: Ken Chase wrote: > 3 of my internet-lifetimes/startups ago, we had this happen when one of the L2 > techs was doing their 'rounds' - but had a backpack on. They swung around and > hit the safety cover on the BRS - which got knocked off. They freaked > out a bit while putting the cover back on... and managed to activate it. If you get a safety cover for your EPO switch, make sure it is the right kind of cover. Following an accidental EPO outage, we got a safety cover that was actually a latch designed to ensure the switch stays pressed until manually reset. We discoverd this when someone tried to demonstrate that it was now a lot harder to accidentally press the EPO. (it wasn't) Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Biscay: Variable 3 or 4. Moderate, occasionally slight. Fair. Moderate or good. From mcole.mailinglists at gmail.com Thu Sep 1 13:40:03 2016 From: mcole.mailinglists at gmail.com (Maxwell Cole) Date: Thu, 1 Sep 2016 09:40:03 -0400 Subject: Arbor Reports 540Gbps "Sustained" Attack In-Reply-To: References: Message-ID: Heya. I can?t speak with any evidence but I do have some infrastructure in Brazil and I can tell you I saw stubbornly persistent packet loss for the past two months. Across at least two tier one backbones. I don?t know anything about 500Gbps but large sustained DDoSes against BR locations for the past two months would not surprise me in the least. Cheers, Max > On Aug 31, 2016, at 3:37 PM, Dennis B wrote: > > https://www.arbornetworks.com/blog/asert/rio-olympics-take-gold-540gbsec-sustained-ddos-attacks/ > > I've used SP Peakflow before and I have my opinions. With all the > intelligence out there about DDoS attacks, DDoS attackers, DDoS tools and > techniques this article leaves me with ton's of questions. > > IE: What industry was the attack target? Was it a single customer or > multiple customers at the same time? What was the attack vector? Was it > multi-vector? What was the duration of the 540Gbps attack? Did you actually > block the attack or did you just report on it from your cloud signaling > alliance aka cloud offering? Could you help explain if the peak of the > attack lasted X minutes, Y hours, Z days? What was the attack targeted > protocol? Was it TCP against TCP or UDP against UDP or UDP against TCP? > > I have to be honest, IDK if Arbor is attempting to claim the largest > recorded DDoS attack in the world cup of DDoS attacks but the fact that > your a local appliance shop. Selling to the global 100 and T1-3 ISPs - I'd > hope for more than a marketing ploy to take the top attack vector. > > Thought I'd ask Nanog if they heard any whispers about this "white > buffalo", which ISPs were Transiting the event, what course of actions were > taken. > > Thanks! From marty at cloudflare.com Thu Sep 1 14:18:15 2016 From: marty at cloudflare.com (Marty Strong) Date: Thu, 1 Sep 2016 15:18:15 +0100 Subject: Optical Wave Providers In-Reply-To: References: Message-ID: <30789629-6F4E-4453-940A-1FC15ABC6754@cloudflare.com> Zayo and Electric lightwave are two options, not sure who owns the fibre in the ground in each case. Regards, Marty Strong -------------------------------------- CloudFlare - AS13335 Network Engineer marty at cloudflare.com +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335 > On 1 Sep 2016, at 00:08, tim at 29lagrange.com wrote: > > I have been looking at optical wave carriers for some long haul 1G/10G across the US. All to major cities and well known POP's. > I am finding that there are not a lot of carriers who are offering wave services, usually just ethernet/MPLS. > Particularly across the North west. > Can someone shed some light on who some of the bigger carriers are and any challenges you have encountered with services like this? > Who actually owns the fiber across the US? > > Thanks > > Tim From jayhanke at gmail.com Thu Sep 1 14:42:18 2016 From: jayhanke at gmail.com (Jay Hanke) Date: Thu, 1 Sep 2016 09:42:18 -0500 Subject: Optical Wave Providers In-Reply-To: References: Message-ID: There are lots of national carriers in the US. A much smaller number of those carriers actually own the fiber cables. There are a handful (Zayo, Level3, CenturyLink, Windstream, Earthlink, Verizon) that have very large national, or semi-national foot prints. The carriers frequently trade and lease strands of fiber from each other to create a national network. Be careful on the commodity routes diversity wise. There are a lot of places with 20+ carriers in the same cable (or duct) each claiming to own the route. Jay On Wed, Aug 31, 2016 at 6:08 PM, wrote: > I have been looking at optical wave carriers for some long haul 1G/10G > across the US. All to major cities and well known POP's. > I am finding that there are not a lot of carriers who are offering wave > services, usually just ethernet/MPLS. > Particularly across the North west. > Can someone shed some light on who some of the bigger carriers are and any > challenges you have encountered with services like this? > Who actually owns the fiber across the US? > > Thanks > > Tim From rod.beck at unitedcablecompany.com Thu Sep 1 15:40:06 2016 From: rod.beck at unitedcablecompany.com (Rod Beck) Date: Thu, 1 Sep 2016 15:40:06 +0000 Subject: Optical Wave Providers In-Reply-To: References: , Message-ID: It is a good point about the conduit diversity. Lots of guys in the Wiltel conduit, for example. Right now there are a lot of new regional fiber optic networks and also some new dark fiber networks (one is connecting all the Trans-Atlantic landing stations and telecom hotels in New Jersey). There are always new networks emerging offer lower latency, new physical diversity or just new interesting routes. - R. ________________________________ From: NANOG on behalf of Jay Hanke Sent: Thursday, September 1, 2016 4:42 PM To: tim at 29lagrange.com Cc: nanog at nanog.org Subject: Re: Optical Wave Providers There are lots of national carriers in the US. A much smaller number of those carriers actually own the fiber cables. There are a handful (Zayo, Level3, CenturyLink, Windstream, Earthlink, Verizon) that have very large national, or semi-national foot prints. The carriers frequently trade and lease strands of fiber from each other to create a national network. Be careful on the commodity routes diversity wise. There are a lot of places with 20+ carriers in the same cable (or duct) each claiming to own the route. Jay On Wed, Aug 31, 2016 at 6:08 PM, wrote: > I have been looking at optical wave carriers for some long haul 1G/10G > across the US. All to major cities and well known POP's. > I am finding that there are not a lot of carriers who are offering wave > services, usually just ethernet/MPLS. > Particularly across the North west. > Can someone shed some light on who some of the bigger carriers are and any > challenges you have encountered with services like this? > Who actually owns the fiber across the US? > > Thanks > > Tim From mpetach at netflight.com Thu Sep 1 21:45:12 2016 From: mpetach at netflight.com (Matthew Petach) Date: Thu, 1 Sep 2016 14:45:12 -0700 Subject: Optical Wave Providers In-Reply-To: References: Message-ID: (Speaking purely for myself, and thoroughly demonstrating my relative ignorance on the topic, but also opening up an opportunity to become better educated...) You may find that optical providers don't really want to mix 1G/10G waves in on systems that are running Nx100G waves on the fiber. With 100G coherent systems, optical dispersion compensation is no longer necessary, as the DSP on the receiving side takes care of it. The 10G waves, on the other hand, would need dispersion compensation along the run, which increases the cost of building and maintaining such a system, because they'd have to peel off your 10G waves to periodically do dispersion compensation, then add them back in. It's far more cost effective for long haul providers to carry the traffic as native 100G, and provide the lower-speed handoff to you at the endpoints via ethernet framing or MPLS. It's not so much a matter of "not enough people owning fiber across the US" as "Oh geez, you want us to run our system in an inefficient and uneconomical mode? Uh...maybe you could call those other guys down the street instead." I suspect that if you ran an experiment by calling for quotes and availability for 100G waves between your endpoints you'd find more availability for 100G waves than 10G or 1G native waves. (I'm half hoping to get a flurry of replies telling me I'm completely wrong, and then explaining the real issues to me. If nobody replies, it might mean I'm not entirely wrong). Thanks! Matt On Wed, Aug 31, 2016 at 4:08 PM, wrote: > I have been looking at optical wave carriers for some long haul 1G/10G > across the US. All to major cities and well known POP's. > I am finding that there are not a lot of carriers who are offering wave > services, usually just ethernet/MPLS. > Particularly across the North west. > Can someone shed some light on who some of the bigger carriers are and any > challenges you have encountered with services like this? > Who actually owns the fiber across the US? > > Thanks > > Tim > From swmike at swm.pp.se Fri Sep 2 04:32:22 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Fri, 2 Sep 2016 06:32:22 +0200 (CEST) Subject: Optical Wave Providers In-Reply-To: References: Message-ID: On Wed, 31 Aug 2016, tim at 29lagrange.com wrote: > I have been looking at optical wave carriers for some long haul 1G/10G across > the US. You probably should describe what you mean by "optical wave". If you mean "I want bit-transparent capacity with grey light handoff, that is not overbooked", then that's not really "optical wave". It will probably do what you want though (if I guess what it is you're looking for). I have had good historical success by asking for STM64/OC192 and then run 10GBASE-LW (WAN-PHY) over it. If you want the provider to treat your bits as bits and not packets, don't even tell them you're running packets. If they're providing OC192, they don't need to know. Don't even give them the chance to do the wrong thing by telling them you're running WAN-PHY over it. They might configure their system to support WAN-PHY and all of a sudden their transponders/muxponders might now understand the packets you're running over OC192 and CRC check them and drop them without you noticing. They don't need to know, so don't tell them. -- Mikael Abrahamsson email: swmike at swm.pp.se From mmg at transtelco.net Fri Sep 2 05:07:33 2016 From: mmg at transtelco.net (=?UTF-8?Q?Manuel_Mar=C3=ADn?=) Date: Thu, 1 Sep 2016 23:07:33 -0600 Subject: Software for tracking network related projects and activities Message-ID: Dear Nanog community We are currently using RT for tracking tasks related to network operations like BGP configuration change requests, circuit/ports activation, support tickets, etc, but when trying to track multiple activities that involve multiple departments, the RT (Request Tracker) system does not provide all the tools as a project/tasks management system. I was wondering if you can share what you use for tracking network related projects and activities. Thank you in advance Regards From mpalmer at hezmatt.org Fri Sep 2 06:41:22 2016 From: mpalmer at hezmatt.org (Matt Palmer) Date: Fri, 2 Sep 2016 16:41:22 +1000 Subject: Software for tracking network related projects and activities In-Reply-To: References: Message-ID: <20160902064122.GN4869@hezmatt.org> On Thu, Sep 01, 2016 at 11:07:33PM -0600, Manuel Mar?n wrote: > We are currently using RT for tracking tasks related to network operations > like BGP configuration change requests, circuit/ports activation, support > tickets, etc, but when trying to track multiple activities that involve > multiple departments, the RT (Request Tracker) system does not provide all > the tools as a project/tasks management system. I was wondering if you can > share what you use for tracking network related projects and activities. RT and a couple of custom fields always worked fine for me and my org. - Matt From tom at ninjabadger.net Fri Sep 2 10:09:54 2016 From: tom at ninjabadger.net (Tom Hill) Date: Fri, 2 Sep 2016 11:09:54 +0100 Subject: Optical Wave Providers In-Reply-To: References: Message-ID: <6b5a1d5e-ff8e-a82d-a3ff-899bd213cfcc@ninjabadger.net> On 01/09/16 22:45, Matthew Petach wrote: > (I'm half hoping to get a flurry of replies telling me > I'm completely wrong, and then explaining the real > issues to me. If nobody replies, it might mean I'm > not entirely wrong). You were not wrong on any particular point, but I don't think you may have heard about muxponders previously. I *think* technically it's TDM'd on the coloured side, but I'm not sure... ODU/OTU stuff is somewhat removed from $dayjob. Someone might want to correct me. Have a look at the data sheet for the Ciena WaveServer though; it should give a good idea of how this fits together. Of course, I'm sure more comprehensive muxponder solutions are available for the [inter]national carriers. :) -- Tom From joshbaird at gmail.com Fri Sep 2 12:48:19 2016 From: joshbaird at gmail.com (Josh Baird) Date: Fri, 2 Sep 2016 08:48:19 -0400 Subject: Software for tracking network related projects and activities In-Reply-To: References: Message-ID: JIRA works great for us. On Fri, Sep 2, 2016 at 1:07 AM, Manuel Mar?n wrote: > Dear Nanog community > > We are currently using RT for tracking tasks related to network operations > like BGP configuration change requests, circuit/ports activation, support > tickets, etc, but when trying to track multiple activities that involve > multiple departments, the RT (Request Tracker) system does not provide all > the tools as a project/tasks management system. I was wondering if you can > share what you use for tracking network related projects and activities. > > Thank you in advance > > Regards > From chrisdunn1 at gmail.com Fri Sep 2 03:08:03 2016 From: chrisdunn1 at gmail.com (Chris Dunn) Date: Thu, 1 Sep 2016 22:08:03 -0500 Subject: Don't press the big red buttom on the wall! Message-ID: On Mon, Aug 29, 2016 at 4:51 PM, Sean Donelan wrote: > > See that big red button on the wall under the sign "Do Not Push This > Button!".... > > > This is going to date me, well, because it happened in my high school years mid 1970's... but my best power off story was when in senior year we were working on the school Sperry/Univac Solid State Systems 90 mainframe. ( https://en.wikipedia.org/wiki/UNIVAC_Solid_State) Well this thing used a Mil-Spec 4K DRUM as its main memory. Now being mil-spec means the motor to spin the drum is 400 cycle AC. Where do you get 400 cycle AC in a 60 cycle world? Well you install a 60 cycle motor in the base of this monstrosity, and connect it via a fan belt to a 400 cycle AC generator that feeds the drum. We came in to class one day to warm the Univac up (it took about 30 minutes to fully power up) and noticed the machine being very quiet when we applied power. We eventually tracked it down to no power to the drum and it not spinning. And the reason the drum was not spinning? The fan belt had broken on the motor/generator set. :) Lucky, this was a Technical/Vocational high school, and we ran down to the auto repair class and bummed a new belt off them. :) ---- Then there was the time an engineer I worked with pressed the lamp test button on the huge engineering console display for a Burroughs B6700 room sized mainframe (like 3 foot by 3 foot bank of incandescent register and status displays) and the inrush caused the entire machine shut down.... but that's another story. ;) Chris Dunn Data Center Operations ServerCentral From bicknell at ufp.org Fri Sep 2 15:29:09 2016 From: bicknell at ufp.org (Leo Bicknell) Date: Fri, 2 Sep 2016 08:29:09 -0700 Subject: Optical Wave Providers In-Reply-To: References: Message-ID: <20160902152908.GA2445@ussenterprise.ufp.org> You're not wrong, but you're not right for two reasons. I believe the OP really wants a "transparent" service. It could be a true wave, but it could also be a 10G channel muxed on a 100G service. The properties they probably really care about are a raw bitstream and guaranteed bandwidth. In the world of marketing and sales speek with carriers this is a "wavelength" service, even if it's not a true wavelength. Indeed, these services are often made up of different underlying services end to end, a dark fiber tail, a Nx10G local transport, muxed on to an Nx100G long haul transport, and in reverse on the other end. On the technical side of things there are plenty of carriers with Nx10G systems across country that have not upgraded them to Nx100G for any number of reasons, often they are 40-80% full of paying customers and profitable. They are quite happy to sell an additional 10G wave and get more money out of their sunk cost. Indeed, if you want a wave on the right path (read, paid for, with plenty of free capacity) you can get waves for rock bottom prices. To the OP's original point, major carriers that offer this class of service include Level 3, CenturyLink, Zayo, and XO. Depending on location there are regional carriers like LightTower, specialzed providers IX Reach, or possibly even your favorite colocation provider like Equinix or CoreSite. There's probably at least 30 more, many don't advertise these services widely (low margin, requires clued customer) but if you ask they are available. -- Leo Bicknell - bicknell at ufp.org PGP keys at http://www.ufp.org/~bicknell/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 811 bytes Desc: not available URL: From jeremy at vcn.com Fri Sep 2 16:24:44 2016 From: jeremy at vcn.com (Jeremy Malli) Date: Fri, 2 Sep 2016 16:24:44 +0000 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU Message-ID: I'm hoping somebody on the list has a recommendation for an outdoor ADSL2+/VDSL/G.Fast NIU. Been doing so some research into this and have come up empty so far. My thinking is that by housing the DSL CPE outside the residence in an enclosure we can reduce the issues with IW (since we would only need a small jumper from the LEC handoff to the NIU) and also gain access to the DSL CPE remotely for management and troubleshooting. We would then hand off ethernet to the customer using existing wiring or running cat5. Interested in how this problem may have already been addressed in the provider community. Thanks, --------------------- Jeremy Malli jeremy at vcn.com From cscora at apnic.net Fri Sep 2 18:01:34 2016 From: cscora at apnic.net (Routing Analysis Role Account) Date: Sat, 3 Sep 2016 04:01:34 +1000 (AEST) Subject: Weekly Routing Table Report Message-ID: <20160902180134.780CDA7544@thyme.apnic.net> This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats at lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith . Routing Table Report 04:00 +10GMT Sat 03 Sep, 2016 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 608392 Prefixes after maximum aggregation (per Origin AS): 219785 Deaggregation factor: 2.77 Unique aggregates announced (without unneeded subnets): 297057 Total ASes present in the Internet Routing Table: 54732 Prefixes per ASN: 11.12 Origin-only ASes present in the Internet Routing Table: 36408 Origin ASes announcing only one prefix: 15422 Transit ASes present in the Internet Routing Table: 6497 Transit-only ASes present in the Internet Routing Table: 177 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 54 Max AS path prepend of ASN ( 55644) 51 Prefixes from unregistered ASNs in the Routing Table: 60 Unregistered ASNs in the Routing Table: 15 Number of 32-bit ASNs allocated by the RIRs: 15289 Number of 32-bit ASNs visible in the Routing Table: 11827 Prefixes from 32-bit ASNs in the Routing Table: 46954 Number of bogon 32-bit ASNs visible in the Routing Table: 56 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 380 Number of addresses announced to Internet: 2824303844 Equivalent to 168 /8s, 87 /16s and 116 /24s Percentage of available address space announced: 76.3 Percentage of allocated address space announced: 76.3 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.2 Total number of prefixes smaller than registry allocations: 197819 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 156248 Total APNIC prefixes after maximum aggregation: 43060 APNIC Deaggregation factor: 3.63 Prefixes being announced from the APNIC address blocks: 169530 Unique aggregates announced from the APNIC address blocks: 68826 APNIC Region origin ASes present in the Internet Routing Table: 5189 APNIC Prefixes per ASN: 32.67 APNIC Region origin ASes announcing only one prefix: 1158 APNIC Region transit ASes present in the Internet Routing Table: 939 Average APNIC Region AS path length visible: 4.5 Max APNIC Region AS path length visible: 54 Number of APNIC region 32-bit ASNs visible in the Routing Table: 2347 Number of APNIC addresses announced to Internet: 759491140 Equivalent to 45 /8s, 68 /16s and 234 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 183404 Total ARIN prefixes after maximum aggregation: 89371 ARIN Deaggregation factor: 2.05 Prefixes being announced from the ARIN address blocks: 189103 Unique aggregates announced from the ARIN address blocks: 88251 ARIN Region origin ASes present in the Internet Routing Table: 16207 ARIN Prefixes per ASN: 11.67 ARIN Region origin ASes announcing only one prefix: 5710 ARIN Region transit ASes present in the Internet Routing Table: 1713 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 1433 Number of ARIN addresses announced to Internet: 1105903584 Equivalent to 65 /8s, 234 /16s and 191 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 145404 Total RIPE prefixes after maximum aggregation: 71686 RIPE Deaggregation factor: 2.03 Prefixes being announced from the RIPE address blocks: 155575 Unique aggregates announced from the RIPE address blocks: 96261 RIPE Region origin ASes present in the Internet Routing Table: 18117 RIPE Prefixes per ASN: 8.59 RIPE Region origin ASes announcing only one prefix: 7831 RIPE Region transit ASes present in the Internet Routing Table: 3020 Average RIPE Region AS path length visible: 4.4 Max RIPE Region AS path length visible: 27 Number of RIPE region 32-bit ASNs visible in the Routing Table: 5030 Number of RIPE addresses announced to Internet: 708253568 Equivalent to 42 /8s, 55 /16s and 23 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 61769 Total LACNIC prefixes after maximum aggregation: 12381 LACNIC Deaggregation factor: 4.99 Prefixes being announced from the LACNIC address blocks: 77242 Unique aggregates announced from the LACNIC address blocks: 37237 LACNIC Region origin ASes present in the Internet Routing Table: 2476 LACNIC Prefixes per ASN: 31.20 LACNIC Region origin ASes announcing only one prefix: 551 LACNIC Region transit ASes present in the Internet Routing Table: 582 Average LACNIC Region AS path length visible: 4.8 Max LACNIC Region AS path length visible: 22 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 2764 Number of LACNIC addresses announced to Internet: 170489920 Equivalent to 10 /8s, 41 /16s and 120 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-265628 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 14552 Total AfriNIC prefixes after maximum aggregation: 3277 AfriNIC Deaggregation factor: 4.44 Prefixes being announced from the AfriNIC address blocks: 16562 Unique aggregates announced from the AfriNIC address blocks: 6170 AfriNIC Region origin ASes present in the Internet Routing Table: 732 AfriNIC Prefixes per ASN: 22.63 AfriNIC Region origin ASes announcing only one prefix: 172 AfriNIC Region transit ASes present in the Internet Routing Table: 180 Average AfriNIC Region AS path length visible: 4.6 Max AfriNIC Region AS path length visible: 21 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 253 Number of AfriNIC addresses announced to Internet: 79780096 Equivalent to 4 /8s, 193 /16s and 89 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5541 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3524 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 3208 11146 1129 KIXS-AS-KR Korea Telecom, KR 17974 2943 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2679 1494 527 BSNL-NIB National Internet Backbone, IN 9808 2144 8781 42 CMNET-GD Guangdong Mobile Communication 4755 2057 429 227 TATACOMM-AS TATA Communications formerl 4808 1761 2291 524 CHINA169-BJ China Unicom Beijing Provin 45899 1570 1227 93 VNPT-AS-VN VNPT Corp, VN 24560 1538 508 220 AIRTELBROADBAND-AS-AP Bharti Airtel Ltd Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 22773 3508 2964 146 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 6389 2201 3671 41 BELLSOUTH-NET-BLK - BellSouth.net Inc., 18566 2195 405 110 MEGAPATH5-US - MegaPath Corporation, US 20115 1936 1977 403 CHARTER-NET-HKY-NC - Charter Communicat 30036 1767 343 251 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 209 1702 5073 655 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 6983 1677 848 227 ITCDELTA - Earthlink, Inc., US 16509 1380 2530 454 AMAZON-02 - Amazon.com, Inc., US 7018 1370 20093 1008 ATT-INTERNET4 - AT&T Services, Inc., US 11492 1284 229 621 CABLEONE - CABLE ONE, INC., US Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3329 169 15 ALJAWWALSTC-AS , SA 20940 2799 1071 1984 AKAMAI-ASN1 , US 34984 1975 327 356 TELLCOM-AS , TR 12479 1340 1018 46 UNI2-AS , ES 13188 1207 98 71 BANKINFORM-AS , UA 8551 1203 377 46 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 6849 1147 355 21 UKRTELNET , UA 8402 1051 544 15 CORBINA-AS Russia, RU 9198 916 352 25 KAZTELECOM-AS , KZ 12389 898 1087 407 ROSTELECOM-AS , RU Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 10620 3476 540 171 Telmex Colombia S.A., CO 8151 2272 3385 555 Uninet S.A. de C.V., MX 7303 1542 953 243 Telecom Argentina S.A., AR 6503 1396 437 55 Axtel, S.A.B. de C.V., MX 11830 1327 367 57 Instituto Costarricense de Electricidad 6147 1081 377 27 Telefonica del Peru S.A.A., PE 7738 994 1882 40 Telemar Norte Leste S.A., BR 3816 961 463 210 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 11172 907 125 76 Alestra, S. de R.L. de C.V., MX 28573 896 2176 161 CLARO S.A., BR Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1182 402 49 LINKdotNET-AS, EG 36903 684 344 115 MT-MPLS, MA 37611 645 48 2 Afrihost, ZA 36992 552 1373 25 ETISALAT-MISR, EG 8452 527 1472 15 TE-AS TE-AS, EG 37492 384 250 69 ORANGE-TN, TN 24835 344 610 16 RAYA-AS, EG 29571 301 37 12 CITelecom-AS, CI 15399 295 35 6 WANANCHI-KE, KE 2018 265 327 74 TENET-1, ZA Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5541 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3524 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 22773 3508 2964 146 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3476 540 171 Telmex Colombia S.A., CO 39891 3329 169 15 ALJAWWALSTC-AS , SA 4766 3208 11146 1129 KIXS-AS-KR Korea Telecom, KR 17974 2943 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 20940 2799 1071 1984 AKAMAI-ASN1 , US 9829 2679 1494 527 BSNL-NIB National Internet Backbone, IN 8151 2272 3385 555 Uninet S.A. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 22773 3508 3362 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 39891 3329 3314 ALJAWWALSTC-AS , SA 10620 3476 3305 Telmex Colombia S.A., CO 7545 3524 3274 TPG-INTERNET-AP TPG Telecom Limited, AU 17974 2943 2865 TELKOMNET-AS2-AP PT Telekomunikasi Indo 6389 2201 2160 BELLSOUTH-NET-BLK - BellSouth.net Inc., 9829 2679 2152 BSNL-NIB National Internet Backbone, IN 9808 2144 2102 CMNET-GD Guangdong Mobile Communication 18566 2195 2085 MEGAPATH5-US - MegaPath Corporation, US 4766 3208 2079 KIXS-AS-KR Korea Telecom, KR Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65001 PRIVATE 5.143.176.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.200.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.208.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.216.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65000 PRIVATE 31.219.177.0/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65000 PRIVATE 31.219.177.128/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65512 PRIVATE 45.252.244.0/24 45899 VNPT-AS-VN VNPT Corp, VN 65512 PRIVATE 45.252.245.0/24 45899 VNPT-AS-VN VNPT Corp, VN Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 23.249.144.0/20 40430 COLO4JAX-AS - colo4jax, LLC, US 27.100.7.0/24 56096 UNKNOWN 41.73.1.0/24 37004 -Reserved AS-, ZZ 41.73.2.0/24 37004 -Reserved AS-, ZZ 41.73.3.0/24 37004 -Reserved AS-, ZZ 41.73.4.0/24 37004 -Reserved AS-, ZZ 41.73.5.0/24 37004 -Reserved AS-, ZZ 41.73.6.0/24 37004 -Reserved AS-, ZZ 41.73.7.0/24 37004 -Reserved AS-, ZZ 41.73.8.0/24 37004 -Reserved AS-, ZZ Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:16 /9:13 /10:36 /11:101 /12:267 /13:516 /14:1052 /15:1768 /16:13136 /17:7811 /18:12992 /19:25305 /20:38539 /21:40208 /22:67597 /23:59424 /24:337949 /25:568 /26:574 /27:386 /28:53 /29:32 /30:14 /31:1 /32:34 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 39891 2896 3329 ALJAWWALSTC-AS , SA 22773 2735 3508 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2087 2195 MEGAPATH5-US - MegaPath Corporation, US 30036 1581 1767 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 6389 1424 2201 BELLSOUTH-NET-BLK - BellSouth.net Inc., 10620 1394 3476 Telmex Colombia S.A., CO 6983 1327 1677 ITCDELTA - Earthlink, Inc., US 34984 1258 1975 TELLCOM-AS , TR 11492 1186 1284 CABLEONE - CABLE ONE, INC., US 13188 993 1207 BANKINFORM-AS , UA Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1619 2:766 4:22 5:2165 6:31 8:983 12:1802 13:42 14:1753 15:45 16:2 17:94 18:126 20:50 22:1 23:1696 24:1793 27:2327 31:1781 32:85 33:4 35:5 36:309 37:2307 38:1251 39:36 40:95 41:2877 42:454 43:1835 44:46 45:2136 46:2570 47:88 49:1207 50:906 51:13 52:540 54:344 55:8 56:7 57:42 58:1641 59:1033 60:381 61:1821 62:1512 63:1909 64:4548 65:2192 66:4230 67:2230 68:1124 69:3258 70:1278 71:517 72:2043 74:2545 75:398 76:411 77:1373 78:1244 79:894 80:1315 81:1402 82:982 83:712 84:854 85:1580 86:485 87:1110 88:550 89:2070 90:215 91:6068 92:953 93:2364 94:2456 95:2478 96:529 97:338 98:944 99:90 100:77 101:1138 103:12145 104:2485 105:126 106:437 107:1342 108:660 109:2261 110:1276 111:1636 112:1049 113:1137 114:1054 115:1683 116:1642 117:1552 118:2043 119:1580 120:926 121:1118 122:2262 123:2016 124:1562 125:1842 128:651 129:455 130:412 131:1381 132:607 133:176 134:484 135:144 136:383 137:402 138:1863 139:419 140:576 141:461 142:662 143:961 144:739 145:163 146:938 147:655 148:1387 149:528 150:659 151:873 152:650 153:295 154:672 155:942 156:524 157:504 158:389 159:1204 160:506 161:732 162:2393 163:562 164:754 165:1101 166:323 167:1160 168:2092 169:664 170:1882 171:252 172:673 173:1697 174:754 175:664 176:1733 177:4027 178:2220 179:1151 180:2144 181:1834 182:2045 183:985 184:826 185:7279 186:3128 187:2115 188:2169 189:1760 190:7679 191:1262 192:9021 193:5713 194:4455 195:3854 196:1732 197:1141 198:5557 199:5752 200:7164 201:3655 202:10107 203:9841 204:4504 205:2737 206:2962 207:3104 208:4015 209:3869 210:3880 211:2051 212:2739 213:2333 214:853 215:71 216:5833 217:1977 218:797 219:613 220:1642 221:884 222:693 223:1286 End of report From josh at kyneticwifi.com Fri Sep 2 23:37:21 2016 From: josh at kyneticwifi.com (Josh Reynolds) Date: Fri, 2 Sep 2016 18:37:21 -0500 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU In-Reply-To: References: Message-ID: Check with Calix or Ciena. On Sep 2, 2016 11:27 AM, "Jeremy Malli" wrote: > I'm hoping somebody on the list has a recommendation for an outdoor > ADSL2+/VDSL/G.Fast NIU. Been doing so some research into this and have > come up empty so far. > > > My thinking is that by housing the DSL CPE outside the residence in an > enclosure we can reduce the issues with IW (since we would only need a > small jumper from the LEC handoff to the NIU) and also gain access to the > DSL CPE remotely for management and troubleshooting. We would then hand > off ethernet to the customer using existing wiring or running cat5. > > > Interested in how this problem may have already been addressed in the > provider community. > > > Thanks, > > > --------------------- > > Jeremy Malli > > jeremy at vcn.com > From rnicklas at verizon.net Sat Sep 3 16:56:29 2016 From: rnicklas at verizon.net (Randolph Nicklas) Date: Sat, 03 Sep 2016 12:56:29 -0400 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU In-Reply-To: References: Message-ID: <4BD4FDCF-F1C1-4E62-9993-E32D40B22499@verizon.net> Adtran. Randy > On Sep 2, 2016, at 19:37, Josh Reynolds wrote: > > Check with Calix or Ciena. > > On Sep 2, 2016 11:27 AM, "Jeremy Malli" wrote: > >> I'm hoping somebody on the list has a recommendation for an outdoor >> ADSL2+/VDSL/G.Fast NIU. Been doing so some research into this and have >> come up empty so far. >> >> >> My thinking is that by housing the DSL CPE outside the residence in an >> enclosure we can reduce the issues with IW (since we would only need a >> small jumper from the LEC handoff to the NIU) and also gain access to the >> DSL CPE remotely for management and troubleshooting. We would then hand >> off ethernet to the customer using existing wiring or running cat5. >> >> >> Interested in how this problem may have already been addressed in the >> provider community. >> >> >> Thanks, >> >> >> --------------------- >> >> Jeremy Malli >> >> jeremy at vcn.com >> From v.bajpai at jacobs-university.de Mon Sep 5 12:36:35 2016 From: v.bajpai at jacobs-university.de (Bajpai, Vaibhav) Date: Mon, 5 Sep 2016 12:36:35 +0000 Subject: measuring web similarity from dual-stacked hosts Message-ID: Dear NANOG, Measuring Web Similarity from Dual-stacked Hosts ------------------------------------------------ How similar are the webpages accessed over IPv6 to their IPv4 counterparts? ? In situations where the content is dissimilar over IPv4 and IPv6, what factors contribute to the dissimilarity? To answer ^ we developed a tool (simweb) and deployed it on 80 geographically distributed dual-stacked SamKnows probes. A paper presenting results from the collected dataset got accepted recently. We just released the tool and the paper [a]. Thought to share it along. [a] http://goo.gl/sAsDcG Feedback most welcome! You may recall a presentation of this work at RIPE 72 [b]. [b] https://ripe72.ripe.net/archives/video/126 Abstract -------- We compare the similarity of webpages delivered over IPv4 and IPv6. Using the SamKnows web performance (webget) test, we implemented an extension (simweb) that allows us to measure the similarity of webpages. The simweb test measures against ALEXA top 100 dual-stacked websites from 80 SamKnows probes connected to dual-stacked networks representing 58 different ASes. Using a two months-long dataset we show that 14% of these dual-stacked websites exhibit a dissimilarity in the number of fetched webpage elements, with 94% of them exhibiting a dissimilarity in their size. We show that 6% of these websites announce AAAA entries in the DNS but no content is delivered over IPv6 when an HTTP request is made. We also noticed several cases where not all webpage elements (such as images, javascript and CSS) of a dual-stacked website are available over IPv6. We show that 27% of the dual-stacked websites have some fraction of webpage elements that fail over IPv6, with 9% of the websites having more than 50% webpage elements that fail over IPv6. We perform a causality analysis and also identify sources for these failing elements. We show that 12% of these websites have more than 50% webpage elements that belong to the same origin source and fail over IPv6. Failure rates are largely affected by DNS resolution error on images, javascript and CSS content delivered from both same-origin and cross-origin sources. These failures tend to cripple experience for users behind an IPv6-only network and a quantification of failure cases may help improve IPv6 adoption on the Internet. -- Vaibhav =================================== Vaibhav Bajpai www.vaibhavbajpai.com Postdoctoral Researcher Jacobs University Bremen, Germany =================================== -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From benno at NLnetLabs.nl Tue Sep 6 12:49:13 2016 From: benno at NLnetLabs.nl (Benno Overeinder) Date: Tue, 6 Sep 2016 14:49:13 +0200 Subject: Last call for presentations and Draft programme for RIPE 73 Message-ID: Colleagues, A list of currently accepted RIPE 73 presentations is now published at: https://ripe73.ripe.net/programme/meeting-plan/draft-programme/ There are still few slots remaining for a final RIPE 73 programme and RIPE Programme Committee will accept new proposals until 25 September 2016. This is our last call for you to submit your proposals. You can find the CFP for RIPE 73 below, or at https://ripe73.ripe.net/submit-topic/cfp/, for your proposals for plenary session presentations, tutorials, workshops, BoFs (Birds of a Feather sessions) and lightning talks. Please also note that speakers do not receive any extra reduction or funding towards the meeting fee at the RIPE Meetings. Kind regards, Benno Overeinder RIPE PC Chair https://www.ripe.net/participate/meetings/ripe-meetings/pc -------------------->>><<<-------------------- Call for Presentations A RIPE Meeting is an open event where Internet Service Providers, network operators and other interested parties get together. Although the meeting is mostly technical, it is also a chance for people to meet and network with others in their field. RIPE 73 will take place from 24-28 October 2016 in Madrid, Spain. The RIPE Programme Committee (PC) is now seeking content proposals from the RIPE community for the plenary sessions, BoFs (Birds of a Feather sessions), panels, workshops, tutorials and lightning talks at RIPE 73. See the full descriptions of the different presentation formats, https://ripe73.ripe.net/submit-topic/presentation-formats/. Proposals for plenary sessions, BoFs, panels, workshops and tutorials must be submitted for full consideration no later than 25 September 2016. Proposals submitted after this date will be considered depending on the remaining available space in the programme. The PC is looking for presentations covering topics of network engineering and operations, including but not limited to: - IPv6 deployment - Managing IPv4 scarcity in operations - Commercial transactions of IPv4 addresses - Data centre technologies - Network and DNS operations - Internet governance and regulatory practices - Network and routing security - Content delivery - Internet peering and mobile data exchange - Internet of Things (IoT) Submissions RIPE Meeting attendees are quite sensitive to keeping presentations non-commercial, and product marketing talks are strongly discouraged. Repeated audience feedback shows that the most successful talks focus on operational experience, research results, or case studies. For example, presenters wishing to describe a commercial solution should focus on the underlying technology and not attempt a product demonstration. Presenters should indicate how much time they will require. In general, the time allocated for the different presentation formats is as follows: - Plenary presentations: 20-25 minutes presentation with 5-10 minutes discussion - Tutorials: up to two hours (Monday morning) - Workshops: one hour (during evening sessions) to two hours (Monday morning) - BoFs: approximately one hour - Lightning talks: 10 minutes The following general requirements apply: - Proposals must be submitted using the meeting submission system, https://ripe73.ripe.net/submit-topic/submission-form/. - Lightning talks should also be submitted using the meeting submission system (https://ripe73.ripe.net/submit-topic/submission-form/) and can be submitted any time up to and including the meeting week. The allocation of lightning talks will be announced on short notice---in some cases on the same day but often one day prior to the time slot allocated. - Presenters who propose a panel or BoF are encouraged to include speakers from several (perhaps even competing) companies and/or a neutral facilitator. - All presentation proposals will only be considered by the PC if they contain at least draft presentation slides (slides may be updated later on). For panels, proposals must contain a clear description, as well as the names of invited panellists, presenters and moderators. - Due to potential technical issues, presenters/panellists should be physically present at the RIPE Meeting. If you have any questions or requests concerning content submissions, please email pc [at] ripe [dot] net. -- Benno J. Overeinder NLnet Labs http://www.nlnetlabs.nl/ From josivan.barbosa01 at gmail.com Tue Sep 6 13:28:21 2016 From: josivan.barbosa01 at gmail.com (Josivan Barbosa) Date: Tue, 6 Sep 2016 10:28:21 -0300 Subject: HUAWEI S6700 - VLANIF MAC ADDRESS Message-ID: Hi all I have a Huawei S6700 and each vlanif has a different mac. Is there any way so that all vlanifs have the same mac address? In brocades switches, for example, all ports have the same mac. Thanks. Josivan Barbosa From dwessels at verisign.com Tue Sep 6 21:33:01 2016 From: dwessels at verisign.com (Wessels, Duane) Date: Tue, 6 Sep 2016 21:33:01 +0000 Subject: Root and ARPA DNSSEC operational message -- signature validity period In-Reply-To: <5F1C5102-CC30-4CAF-9C2D-61AFDF6EC2C7@verisign.com> References: <5F1C5102-CC30-4CAF-9C2D-61AFDF6EC2C7@verisign.com> Message-ID: FYI, this work is now complete. DW > On Aug 30, 2016, at 2:32 PM, Wessels, Duane wrote: > > DNSSEC signatures in the Root and ARPA zones are currently given a validity > period of 10 days. The validity period is being increased to 13 days, per > the recommendations of RSSAC's Report on Root Zone TTLs [1] (aka RSSAC003). > > Note that we are not aware of any cases where the 10-day signature validity > period has caused problems for DNSSEC validators. This is a precautionary > measure designed to accommodate a worst-case scenario. > > This change will be implemented on September 6, 2016. Please feel free > to contact us at RZM at verisign.com with concerns or questions, and to forward > this notice to others who may not have already received it. > > [1] https://www.icann.org/en/system/files/files/rssac-003-root-zone-ttls-21aug15-en.pdf > > DW > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric.kuhnke at gmail.com Wed Sep 7 00:40:06 2016 From: eric.kuhnke at gmail.com (Eric Kuhnke) Date: Tue, 6 Sep 2016 17:40:06 -0700 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU In-Reply-To: References: Message-ID: I think Calix has a fully outdoor version of their 844G VDSL2 modem. The problem you'll run into is the cost of bringing 12VDC through a small hole in the exterior wall (from a small indoor mounted 120VAC to 12VDC power supply) or outdoor code compliant weatherproof 120VAC for the equipment. And the issues that are frequently encountered by cable TV installers who mistakenly drill through a hole through a wall for some RG6 coax and burst somebody's water pipe or sewer pipe. This can drive the cost per building served cost up considerably, if the ISP needs to eat all or a portion of the cost of bringing electrical service to the outdoor mounting location. On Fri, Sep 2, 2016 at 9:24 AM, Jeremy Malli wrote: > I'm hoping somebody on the list has a recommendation for an outdoor > ADSL2+/VDSL/G.Fast NIU. Been doing so some research into this and have > come up empty so far. > > > My thinking is that by housing the DSL CPE outside the residence in an > enclosure we can reduce the issues with IW (since we would only need a > small jumper from the LEC handoff to the NIU) and also gain access to the > DSL CPE remotely for management and troubleshooting. We would then hand > off ethernet to the customer using existing wiring or running cat5. > > > Interested in how this problem may have already been addressed in the > provider community. > > > Thanks, > > > --------------------- > > Jeremy Malli > > jeremy at vcn.com > From jared at puck.nether.net Wed Sep 7 00:55:28 2016 From: jared at puck.nether.net (Jared Mauch) Date: Tue, 6 Sep 2016 20:55:28 -0400 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU In-Reply-To: References: Message-ID: <461CEA0B-AFD7-484C-87C3-C0490FDCC2DD@puck.nether.net> > On Sep 6, 2016, at 8:40 PM, Eric Kuhnke wrote: > > This can drive the cost per building served cost up considerably, if the > ISP needs to eat all or a portion of the cost of bringing electrical > service to the outdoor mounting location. I?m a fan of this device as you can run PoE out to it. Just wish it had space for a splice to be held, as well as excess cable stored. Of course this is fiber vs xDSL, etc.. http://www.balticnetworks.com/mikrotik-fiber-to-copper-converter.html - Jared From carlos at race.com Wed Sep 7 06:47:02 2016 From: carlos at race.com (Carlos Alcantar) Date: Wed, 7 Sep 2016 06:47:02 +0000 Subject: Outdoor ADSL2+/VDSL/G.Fast NIU In-Reply-To: References: , Message-ID: The calix 844g is not an outdoor rated unit just a heads up. Typically now xdsl modems have integrated router and wireless which brings them into indoor rated type of equipment. I'm sure outdoor is out there but probably not something that's off the shelf and highly deployed. ? Carlos Alcantar Race Communications / Race Team Member 1325 Howard Ave. #604, Burlingame, CA. 94010 Phone: +1 415 376 3314 / carlos at race.com / http://www.race.com ________________________________________ From: NANOG on behalf of Eric Kuhnke Sent: Tuesday, September 6, 2016 5:40:06 PM To: nanog at nanog.org list Subject: Re: Outdoor ADSL2+/VDSL/G.Fast NIU I think Calix has a fully outdoor version of their 844G VDSL2 modem. The problem you'll run into is the cost of bringing 12VDC through a small hole in the exterior wall (from a small indoor mounted 120VAC to 12VDC power supply) or outdoor code compliant weatherproof 120VAC for the equipment. And the issues that are frequently encountered by cable TV installers who mistakenly drill through a hole through a wall for some RG6 coax and burst somebody's water pipe or sewer pipe. This can drive the cost per building served cost up considerably, if the ISP needs to eat all or a portion of the cost of bringing electrical service to the outdoor mounting location. On Fri, Sep 2, 2016 at 9:24 AM, Jeremy Malli wrote: > I'm hoping somebody on the list has a recommendation for an outdoor > ADSL2+/VDSL/G.Fast NIU. Been doing so some research into this and have > come up empty so far. > > > My thinking is that by housing the DSL CPE outside the residence in an > enclosure we can reduce the issues with IW (since we would only need a > small jumper from the LEC handoff to the NIU) and also gain access to the > DSL CPE remotely for management and troubleshooting. We would then hand > off ethernet to the customer using existing wiring or running cat5. > > > Interested in how this problem may have already been addressed in the > provider community. > > > Thanks, > > > --------------------- > > Jeremy Malli > > jeremy at vcn.com > From jtk at depaul.edu Wed Sep 7 20:10:06 2016 From: jtk at depaul.edu (John Kristoff) Date: Wed, 7 Sep 2016 15:10:06 -0500 Subject: Security Track @ NANOG 68 Call for Participation Message-ID: <20160907151006.337ae20c@p50.localdomain> [ Apologies if you saw this elsewhere already - jtk ] Friends, colleagues, fellow operators, The network security track, formerly known as the ISP security BoF, may be on the agenda at NANOG 68 in Dallas October 17-19 and if we can put together a reasonable agenda I may be your track facilitator. We not only seek your participation, but we are also interested you contributing to the track content. If you have an idea or if you know someone who might like to present, discuss or conduct a demonstration of something, we want to hear from you. Here is an incomplete list of relevant topics to stir your thoughts: * 802.11 authentication, authorization and accounting * abuse@ management and war stories * BGPSEC and RPKI * DDoS attacks and mitigation * Government regulatory and compliance issues * Internet of Things and security * IPv6 considerations * New security software projects and tools (non-commercial) * Privacy-related issues and topics * The latest in protocol abuse * John From frnkblk at iname.com Wed Sep 7 20:23:30 2016 From: frnkblk at iname.com (Frank Bulk) Date: Wed, 7 Sep 2016 15:23:30 -0500 Subject: Optical transceiver question Message-ID: <001001d20945$b2dc9680$1895c380$@iname.com> We recently purchased some generic optics from a reputable reseller that were marketed to reach 60 km. But what we found, based on the spec sheets, is that it could only reach that distance if the optics were transmitting on the high side of the transmit power range. For example, if the TX range was 0 to +5 dBm and minimum RX power was -20 dB, the designed optical budget should be no more than 20 dB (0 - -20). Based on the wavelength the appropriate loss would be 0.4 dB/km and results in only 50 km, not 60 km. To get 60 km it would need 24 dB of link margin, and that would only be attainable if it was transmitting on the high side, at +4 dBm. Is it an industry practice to market distance based on the hot optics, not on the worst case, which is minimum TX power? Frank From eric.kuhnke at gmail.com Wed Sep 7 20:51:01 2016 From: eric.kuhnke at gmail.com (Eric Kuhnke) Date: Wed, 7 Sep 2016 13:51:01 -0700 Subject: Optical transceiver question In-Reply-To: <001001d20945$b2dc9680$1895c380$@iname.com> References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: What you're saying is if you purchase ten identical optics with the same SKU, and put them on a few hundred meters of coiled SC/UPC to SC/UPC simplex fiber and an optical power meter on the other end, they're showing varying real world Tx powers from between +0 to +5dBm? That's not right at all, they're supposed to be sorted at the factory by their actual optical power output before they have labels put on them. On Wed, Sep 7, 2016 at 1:23 PM, Frank Bulk wrote: > We recently purchased some generic optics from a reputable reseller that > were marketed to reach 60 km. > > But what we found, based on the spec sheets, is that it could only reach > that distance if the optics were transmitting on the high side of the > transmit power range. > > For example, if the TX range was 0 to +5 dBm and minimum RX power was -20 > dB, the designed optical budget should be no more than 20 dB (0 - -20). > Based on the wavelength the appropriate loss would be 0.4 dB/km and results > in only 50 km, not 60 km. To get 60 km it would need 24 dB of link margin, > and that would only be attainable if it was transmitting on the high side, > at +4 dBm. > > Is it an industry practice to market distance based on the hot optics, not > on the worst case, which is minimum TX power? > > Frank > > From jared at puck.nether.net Wed Sep 7 21:26:41 2016 From: jared at puck.nether.net (Jared Mauch) Date: Wed, 7 Sep 2016 17:26:41 -0400 Subject: Optical transceiver question In-Reply-To: <001001d20945$b2dc9680$1895c380$@iname.com> References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: We have seen cases where the patches introduce enough loss to cause a lot of loss. Have you done an OTDR on each link? Jared Mauch > On Sep 7, 2016, at 4:23 PM, Frank Bulk wrote: > > We recently purchased some generic optics from a reputable reseller that > were marketed to reach 60 km. > > But what we found, based on the spec sheets, is that it could only reach > that distance if the optics were transmitting on the high side of the > transmit power range. > > For example, if the TX range was 0 to +5 dBm and minimum RX power was -20 > dB, the designed optical budget should be no more than 20 dB (0 - -20). > Based on the wavelength the appropriate loss would be 0.4 dB/km and results > in only 50 km, not 60 km. To get 60 km it would need 24 dB of link margin, > and that would only be attainable if it was transmitting on the high side, > at +4 dBm. > > Is it an industry practice to market distance based on the hot optics, not > on the worst case, which is minimum TX power? > > Frank From rjacobs at pslightwave.com Wed Sep 7 21:32:59 2016 From: rjacobs at pslightwave.com (Robert Jacobs) Date: Wed, 7 Sep 2016 21:32:59 +0000 Subject: Optical transceiver question In-Reply-To: References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: Not buying fresh veggies here... All optics have about a 5 db range that the vendor will say it is good. The better venders stamp the output power on the optics but not all do this... What he said is to achieve the 60 Km selling point you would have to have all the optic be on the high side of the db TX power... I have never heard of a 60 Km rated optics and it would seem they should be saying 40 to 60 not just 60. It would be nice to say I only want the optics that have an output on the high side and will accept only a 1 db variance but have never seen that in reality. Most are in the middle.. Robert Jacobs | Network Director/Architect Direct:? 832-615-7742 Main:?? 832-615-8000 Fax:??? 713-510-1650 5959 Corporate Dr. Suite 3300; Houston, TX 77036 A Certified Woman-Owned Business 24x7x365 Customer? Support: 832-615-8000 | support at pslightwave.com This electronic message contains information from Phonoscope Lightwave which may be privileged and confidential. The information is intended to be for the use of individual(s) or entity named above. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic message in error, please notify me by telephone or e-mail immediately. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Eric Kuhnke Sent: Wednesday, September 7, 2016 3:51 PM To: nanog at nanog.org list Subject: Re: Optical transceiver question What you're saying is if you purchase ten identical optics with the same SKU, and put them on a few hundred meters of coiled SC/UPC to SC/UPC simplex fiber and an optical power meter on the other end, they're showing varying real world Tx powers from between +0 to +5dBm? That's not right at all, they're supposed to be sorted at the factory by their actual optical power output before they have labels put on them. On Wed, Sep 7, 2016 at 1:23 PM, Frank Bulk wrote: > We recently purchased some generic optics from a reputable reseller > that were marketed to reach 60 km. > > But what we found, based on the spec sheets, is that it could only > reach that distance if the optics were transmitting on the high side > of the transmit power range. > > For example, if the TX range was 0 to +5 dBm and minimum RX power was > -20 dB, the designed optical budget should be no more than 20 dB (0 - -20). > Based on the wavelength the appropriate loss would be 0.4 dB/km and > results in only 50 km, not 60 km. To get 60 km it would need 24 dB of > link margin, and that would only be attainable if it was transmitting > on the high side, at +4 dBm. > > Is it an industry practice to market distance based on the hot optics, > not on the worst case, which is minimum TX power? > > Frank > > From olivier.benghozi at wifirst.fr Wed Sep 7 21:49:28 2016 From: olivier.benghozi at wifirst.fr (Olivier Benghozi) Date: Wed, 7 Sep 2016 23:49:28 +0200 Subject: Optical transceiver question In-Reply-To: <001001d20945$b2dc9680$1895c380$@iname.com> References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: <67A1BA07-5258-4DAC-8BBF-453E33DC614A@wifirst.fr> It's a bit like car fuel efficiency values, even with reputable brands :) In this industry, the number of kms for such optics is a best case approximation of the combination of (most notably) those elements: worst case power budget, capability to deal with chromatic scattering on this length without compensation, with perfect fiber attenuation values without connectors/splicings (by the way, per km attenuation depends of the type of fiber ; so it's really possible to have less than 0.4dB/km even near 1310nm). I confirm it's normal for several optics of the same model to have a large panel of Tx values as soon as they are within the guaranteed Tx range. The actual value of Tx is only guaranteed to be somewhere within the Tx range shown in the specs (and it will vary/lower during the life of the optic). So the power budget of an optic is the value in the worst case. If you cannot OTDR the link (or just put a known source at one side and a power meter at the other side), then you have to estimate: take the length and the attenuation per km for the wavelength, add attenuation values for connectors (crossconnects, patches) and current splicings and so on, take a margin (at least 3dB let's say ?) for later splicings or small defects, and then you obtain the minimum power budget. 10 years later, this page is still relevant: http://www.cisco.com/c/en/us/support/docs/optical-networking/ons-15454-sonet-multiservice-provisioning-platform-mspp/27042-max-att-27042.html Anyway, you must not use the km value for anything else than sorting the answers before looking at the specs :) > Le 7 sept. 2016 ? 22:23, Frank Bulk a ?crit : > > We recently purchased some generic optics from a reputable reseller that > were marketed to reach 60 km. > > But what we found, based on the spec sheets, is that it could only reach > that distance if the optics were transmitting on the high side of the > transmit power range. > > For example, if the TX range was 0 to +5 dBm and minimum RX power was -20 > dB, the designed optical budget should be no more than 20 dB (0 - -20). > Based on the wavelength the appropriate loss would be 0.4 dB/km and results > in only 50 km, not 60 km. To get 60 km it would need 24 dB of link margin, > and that would only be attainable if it was transmitting on the high side, > at +4 dBm. > > Is it an industry practice to market distance based on the hot optics, not > on the worst case, which is minimum TX power? > > Frank > From Daniel.Jameson at tdstelecom.com Wed Sep 7 22:03:07 2016 From: Daniel.Jameson at tdstelecom.com (Jameson, Daniel) Date: Wed, 7 Sep 2016 22:03:07 +0000 Subject: Optical transceiver question In-Reply-To: References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: There are a bunch of variables that impact actual power needed vs. road-miles, number of cross-connects, type of fiber, amount of slack, type of connectors, frequency, dispersion, etc. The km notation simplifies the naming convention. As a general rule 40Km budget 20db, 60km budget 24db, and 80km budget 28db. Best practice is design and deploy to stay a minimum of 4db above the Minimum Input Sensitivity, and 4db below the Max Receive Level to account for the change as the optic and glass as it ages. If you are engineering a 60km fiber length, you'd step up to the next higher optic to account for other losses. 60KM is just a rough number. There is another value to worry about as well. It's the De-Assert Level; an optic will run all the way down to its minimum input sensitivity, sometimes a couple db below. But once it hit's the De-assert level, the optic will stop forwarding what it receives. So there is a little more fudge in the numbers if you're not playing by the rules. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Robert Jacobs Sent: Wednesday, September 07, 2016 3:33 PM To: Eric Kuhnke; nanog at nanog.org list Subject: RE: Optical transceiver question Not buying fresh veggies here... All optics have about a 5 db range that the vendor will say it is good. The better venders stamp the output power on the optics but not all do this... What he said is to achieve the 60 Km selling point you would have to have all the optic be on the high side of the db TX power... I have never heard of a 60 Km rated optics and it would seem they should be saying 40 to 60 not just 60. It would be nice to say I only want the optics that have an output on the high side and will accept only a 1 db variance but have never seen that in reality. Most are in the middle.. Robert Jacobs | Network Director/Architect Direct:? 832-615-7742 Main:?? 832-615-8000 Fax:??? 713-510-1650 5959 Corporate Dr. Suite 3300; Houston, TX 77036 A Certified Woman-Owned Business 24x7x365 Customer? Support: 832-615-8000 | support at pslightwave.com This electronic message contains information from Phonoscope Lightwave which may be privileged and confidential. The information is intended to be for the use of individual(s) or entity named above. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic message in error, please notify me by telephone or e-mail immediately. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Eric Kuhnke Sent: Wednesday, September 7, 2016 3:51 PM To: nanog at nanog.org list Subject: Re: Optical transceiver question What you're saying is if you purchase ten identical optics with the same SKU, and put them on a few hundred meters of coiled SC/UPC to SC/UPC simplex fiber and an optical power meter on the other end, they're showing varying real world Tx powers from between +0 to +5dBm? That's not right at all, they're supposed to be sorted at the factory by their actual optical power output before they have labels put on them. On Wed, Sep 7, 2016 at 1:23 PM, Frank Bulk wrote: > We recently purchased some generic optics from a reputable reseller > that were marketed to reach 60 km. > > But what we found, based on the spec sheets, is that it could only > reach that distance if the optics were transmitting on the high side > of the transmit power range. > > For example, if the TX range was 0 to +5 dBm and minimum RX power was > -20 dB, the designed optical budget should be no more than 20 dB (0 - -20). > Based on the wavelength the appropriate loss would be 0.4 dB/km and > results in only 50 km, not 60 km. To get 60 km it would need 24 dB of > link margin, and that would only be attainable if it was transmitting > on the high side, at +4 dBm. > > Is it an industry practice to market distance based on the hot optics, > not on the worst case, which is minimum TX power? > > Frank > > From eric.kuhnke at gmail.com Wed Sep 7 23:15:47 2016 From: eric.kuhnke at gmail.com (Eric Kuhnke) Date: Wed, 7 Sep 2016 16:15:47 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901101951.usbucglng5itramz@nic.fr> References: <20160901013657.GE4869@hezmatt.org> <20160901101951.usbucglng5itramz@nic.fr> Message-ID: Further update on all known suspicious activity from Wosign: https://wiki.mozilla.org/CA:WoSign_Issues Seriously, what level of malice and/or incompetence does one have to rise to in order to be removed from the Mozilla (and hopefully Microsoft and Chrome) trusted root CA store? Is this not sufficient? On Thu, Sep 1, 2016 at 3:19 AM, Stephane Bortzmeyer wrote: > On Thu, Sep 01, 2016 at 11:36:57AM +1000, > Matt Palmer wrote > a message of 45 lines which said: > > > I'd be surprised if most business continuity people could even name > > their cert provider, > > And they're right because it would be a useless information: without > DANE, *any* CA can issue a certificate for *your* domain, whether you > are a client or not. > From nick at foobar.org Wed Sep 7 23:31:07 2016 From: nick at foobar.org (Nick Hilliard) Date: Thu, 08 Sep 2016 00:31:07 +0100 Subject: Optical transceiver question In-Reply-To: <001001d20945$b2dc9680$1895c380$@iname.com> References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: <57D0A33B.5050106@foobar.org> Frank Bulk wrote: > We recently purchased some generic optics from a reputable reseller that > were marketed to reach 60 km. transceivers don't work like that. They are sold with a specific optical budget, normally rated in dB at a specific wavelength. The km equivalent is usually based on G.652 fibre at an attenuation of 0.25dB per km at 1550nm, or 0.35dB loss at 1330 nm. This means that your optical link needs to fit within the optical budget of your transceivers. If you put a pair of muxes in the middle, you need to subtract the mux- and demux attenuation from your budget. Each patch should be assumed to lose ~0.5dB, each splice should assume 0.1dB and you need to add in a 2dB overhead for loss over time. You need to feed all these figures into an optical budget calculator and see what it comes out with. There are plenty online, some good, some not so good. If you're using a technology which is affect by dispersion, you may need to subtract a couple more dB to compensate for this, or if the link is too long or the technology requires it (10G dwdm or non coherent 100G), you may need to implement optical dispersion compensation. When dealing with optical links of this sort of size, the first and most important thing to start out with is an accurate characterisation of your optical link, end-to-end, at multiple wavelengths, but particularly at 1550nm because that is the reference point for C band WDM. Your provider should provide this as part of the hand-off process. Once you have this data, you should be able to work out what you can support on it. If you don't have this data, you cannot ever know for sure what your optical link can support. For more information, check out Richard Steenbergen's presentations on optical design at https://www.nanog.org/resources/tutorials: Optical Networking 101 (Feb 4 2013) Everything You Always Wanted to Know About Optical Networking (June 13, 2016) These are both excellent introductions to optical design, and it's worth spending the time watching the youtube videos associated with these talks. Nick From julionicolascortes at gmail.com Wed Sep 7 21:25:52 2016 From: julionicolascortes at gmail.com (Nicolas Cortes) Date: Wed, 7 Sep 2016 18:25:52 -0300 Subject: Optical transceiver question In-Reply-To: References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: The typical situation of the vendor is that the link-budget of the transceiver considers the worst scenario for TX and loss of dBs generated by time of operation of the laser, standard attenuation of the fiber, how it changes in how old it is,... in other ways, the calc ispessimistic. In my experience, the transceivers have a variation in TX power from 1-3 dBs in worst cases. In short-haul links doesn't matter very much. In long-haul is something to consider. Nicolas./~ On Wednesday, 7 September 2016, Eric Kuhnke wrote: > What you're saying is if you purchase ten identical optics with the same > SKU, and put them on a few hundred meters of coiled SC/UPC to SC/UPC > simplex fiber and an optical power meter on the other end, they're showing > varying real world Tx powers from between +0 to +5dBm? > > That's not right at all, they're supposed to be sorted at the factory by > their actual optical power output before they have labels put on them. > > On Wed, Sep 7, 2016 at 1:23 PM, Frank Bulk > wrote: > > > We recently purchased some generic optics from a reputable reseller that > > were marketed to reach 60 km. > > > > But what we found, based on the spec sheets, is that it could only reach > > that distance if the optics were transmitting on the high side of the > > transmit power range. > > > > For example, if the TX range was 0 to +5 dBm and minimum RX power was -20 > > dB, the designed optical budget should be no more than 20 dB (0 - -20). > > Based on the wavelength the appropriate loss would be 0.4 dB/km and > results > > in only 50 km, not 60 km. To get 60 km it would need 24 dB of link > margin, > > and that would only be attainable if it was transmitting on the high > side, > > at +4 dBm. > > > > Is it an industry practice to market distance based on the hot optics, > not > > on the worst case, which is minimum TX power? > > > > Frank > > > > > From george.herbert at gmail.com Wed Sep 7 23:39:14 2016 From: george.herbert at gmail.com (George William Herbert) Date: Wed, 7 Sep 2016 16:39:14 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901101951.usbucglng5itramz@nic.fr> References: <20160901013657.GE4869@hezmatt.org> <20160901101951.usbucglng5itramz@nic.fr> Message-ID: > On Sep 1, 2016, at 3:19 AM, Stephane Bortzmeyer wrote: > > On Thu, Sep 01, 2016 at 11:36:57AM +1000, > Matt Palmer wrote > a message of 45 lines which said: > >> I'd be surprised if most business continuity people could even name >> their cert provider, > > And they're right because it would be a useless information: without > DANE, *any* CA can issue a certificate for *your* domain, whether you > are a client or not. It's relevant for a different reason; CA health needs to be monitored, and multiple CAs can (should) be used in case CA A's recognition gets pulled or a catastrophe happens. Having certs from CA B then gets you going either immediately (if you actively use both) or rapidly (if you need to replace certs on web / services front end). Getting new ones from CA B in a hurry can be a major deal. Sent from my iPhone From george.herbert at gmail.com Wed Sep 7 23:44:36 2016 From: george.herbert at gmail.com (George William Herbert) Date: Wed, 7 Sep 2016 16:44:36 -0700 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: <20160901101017.GW4869@hezmatt.org> References: <20160901013657.GE4869@hezmatt.org> <20160901101017.GW4869@hezmatt.org> Message-ID: > On Sep 1, 2016, at 3:10 AM, Matt Palmer wrote: > > How the hell do you get from "the world does not work that way" to "please > pitch me your consulting services"? You appear ignorant of what real DR / resiliency can do, as do your local providers if they said that. I didn't name the company I work for because I'm not advertising, but trying to educate. I'm sorry if the kind of flip answer that it's being done rubbed you the wrong way. Sent from my iPhone From mpalmer at hezmatt.org Thu Sep 8 06:09:43 2016 From: mpalmer at hezmatt.org (Matt Palmer) Date: Thu, 8 Sep 2016 16:09:43 +1000 Subject: Chinese root CA issues rogue/fake certificates In-Reply-To: References: <20160901013657.GE4869@hezmatt.org> <20160901101951.usbucglng5itramz@nic.fr> Message-ID: <20160908060943.GD4869@hezmatt.org> On Wed, Sep 07, 2016 at 04:15:47PM -0700, Eric Kuhnke wrote: > Further update on all known suspicious activity from Wosign: > > https://wiki.mozilla.org/CA:WoSign_Issues > > Seriously, what level of malice and/or incompetence does one have to rise > to in order to be removed from the Mozilla (and hopefully Microsoft and > Chrome) trusted root CA store? Is this not sufficient? At this point, it's pretty clear that WoSign as an operational CA is going to be no more, at least as far as Mozilla is concerned. The number of issues is immense, and nobody on m.d.s.p is arguing in favour of keeping the root (except WoSign). The other major trust stores are completely opaque as to their process, but a root pulled from Mozilla is practically dead in the water. The problem is that just pulling the root is extremely damaging -- to Mozilla, and to the ecosystem. If a root gets pulled, all the sites that are currently using a WoSign-issued cert "stop working". Since plenty of people use WoSign certs (in China, as well as their "free" issuance offering), a lot of sites go dead all at once. Since users cannot stand to not have their dancing kitten gifs, they'll barge through any barrier you put in place, whether that be clicking past warnings or switching to another browser. Mozilla doesn't want to lose (more) market share, and training people to click past security warnings is a really, really dumb move. There are a number of things that could be done to reduce the mess of a pulled root, but many of them involve the cooperation of the CA being pulled, and it's highly unlikely that they'd be in a cooperative mood. The relevant discussion at the moment is around how best to cause WoSign to no longer be trusted, *without* causing collateral damage (or at least minimising it). Certificate Transparency can help, maybe, but CT isn't a live query mechanism, and shipping a giant whitelist of all valid WoSign certs is... large. Honest Achmed had the right idea. - Matt Nit-pickers' corner: Chrome uses the OS trust store; Google doesn't run its own trust store for Chrome, although it does maintain *something* for Android. Chrome has a cert blacklist, and its own EV treatment criteria, but no trust store as such. From alex at alexwacker.com Thu Sep 8 14:13:36 2016 From: alex at alexwacker.com (Alex Wacker) Date: Thu, 8 Sep 2016 10:13:36 -0400 Subject: Google Apps Contact Message-ID: Would appreciate if someone who manages or provides support for google apps could contact me off list.? --? Alex Wacker From selliott at getunwired.com Thu Sep 8 15:52:48 2016 From: selliott at getunwired.com (Shon Elliott) Date: Thu, 8 Sep 2016 15:52:48 +0000 Subject: Pandora Contact Message-ID: <2FC66B6EEF733844895E94A7FEE4FA5207372E96@mbx032-e1-va-4.exch032.serverpod.net> Can someone from Pandora please contact me off-list? Thanks! Kind Regards, Shon Elliott, KK6TOO selliott at getunwired.com From selliott at getunwired.com Thu Sep 8 17:54:42 2016 From: selliott at getunwired.com (Shon Elliott) Date: Thu, 8 Sep 2016 17:54:42 +0000 Subject: Amazon Contact Message-ID: <2FC66B6EEF733844895E94A7FEE4FA5207373EC0@mbx032-e1-va-4.exch032.serverpod.net> Hi everyone, Sorry for having to ask this, but I haven't been able to chase down anyone from Amazon. Can someone from Amazon who might be watching the list who deals with the Amazon Instant Video, FireTV, Music, and other streaming media sections please contact me off-list regarding a serious issue? I would appreciate it very much. Kind Regards, Shon Elliott, KK6TOO selliott at getunwired.com From liam at fedney.org Thu Sep 8 17:59:12 2016 From: liam at fedney.org (L Sean Kennedy) Date: Thu, 8 Sep 2016 13:59:12 -0400 Subject: NANOG 68 Peering Track Call for Participation Message-ID: The NANOG Program Committee is looking for interested presenters for the NANOG 68 Peering Track (10/19/2017). If you are interested in presenting or can suggestion a presenter, please contact the NANOG Program Committee ( nanogpc at nanog.org) or myself. Possible topic areas are listed below, but the PC welcomes other subject related to peering and interconnection. Please note that Peering Personals will continue to be a separate activity at NANOG 68 to be held Monday 10/17 following the program. - Peering Economics and Interconnection models - Measurement and Analysis specific to peering - Alternatives to hot-potato routing - Operations including Peering Databases - Transit as an alternative to peering - 100GbE Interconnection - Validation of routing information - Peering-specific lightning talks (10 minutes or less) Thanks, Sean NANOG Program Committee From swmike at swm.pp.se Thu Sep 8 18:36:08 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Thu, 8 Sep 2016 20:36:08 +0200 (CEST) Subject: Optical transceiver question In-Reply-To: <001001d20945$b2dc9680$1895c380$@iname.com> References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: On Wed, 7 Sep 2016, Frank Bulk wrote: > Is it an industry practice to market distance based on the hot optics, > not on the worst case, which is minimum TX power? No. If this is 1310nm optics with 0.4dB/km budget, the budget figure should be end-of-life figure, ie worst case according to the specs. I don't like the "kilometer" figures, that can be marketed with very optimistic figures. However, if the transceiver says 0 to -5 transmit, if it doesn't transmit 0 to -5 then it's out of spec. I treat the kilometer figure as "marketing", and look only at the optical specifications. So using your figures, if the device doesn't have 0 to -5 out, and can receive error free at -20, then it's out of spec and it should be replaced free of charge. However, if they market 1310nm with 15dB link budget at 60km reach, then I'd consder that false marketing. -- Mikael Abrahamsson email: swmike at swm.pp.se From pshem.k at gmail.com Thu Sep 8 22:12:30 2016 From: pshem.k at gmail.com (Pshem Kowalczyk) Date: Thu, 08 Sep 2016 22:12:30 +0000 Subject: Use of unique local IPv6 addressing rfc4193 Message-ID: Hi, We're looking at rolling out IPv6 to our internal DC infrastructure. Those systems support only our internal network and in the IPv4 world they all live in 'private' space of 10.0.0.0/8. I was wondering if anyone uses the fc00::/7 space for these sort of things or do ppl use a bit of their public IPv6 allocation and manage the security for those ranges? I realise I'd have to use a proxy or NAT66 for the regular outbound connectivity (but we do it already for IPv4 anyway). The truth is that even if we do use something out of our public allocation we're likely to do the same thing (just to be sure that nothing spills out accidentally). So what do you do in this space? kind regards Pshem From marka at isc.org Thu Sep 8 22:26:56 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 09 Sep 2016 08:26:56 +1000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: Your message of "Thu, 08 Sep 2016 22:12:30 +0000." References: Message-ID: <20160908222656.62E5E53946A8@rock.dv.isc.org> In message , Pshem Kowalczyk writes: > Hi, > > We're looking at rolling out IPv6 to our internal DC infrastructure. Those > systems support only our internal network and in the IPv4 world they all > live in 'private' space of 10.0.0.0/8. I was wondering if anyone uses the > fc00::/7 space for these sort of things or do ppl use a bit of their public > IPv6 allocation and manage the security for those ranges? > I realise I'd have to use a proxy or NAT66 for the regular outbound > connectivity (but we do it already for IPv4 anyway). The truth is that even > if we do use something out of our public allocation we're likely to do the > same thing (just to be sure that nothing spills out accidentally). > > So what do you do in this space? > > kind regards > Pshem If you have a NAT you can't prevent things spilling out. The ONLY way to prevent things spilling out is to not connect the network in any shape or form. All NAT does is make it harder to run your network and increases the cost of software development. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From pshem.k at gmail.com Thu Sep 8 23:09:28 2016 From: pshem.k at gmail.com (Pshem Kowalczyk) Date: Thu, 08 Sep 2016 23:09:28 +0000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: <20160908222656.62E5E53946A8@rock.dv.isc.org> References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: With NAT I have a single entry/exit point to those infrastructure subnets which can be easily policed. If I give them public IPs then they're routable and potentially can reach the internet via devices that don't police the traffic. My real question is does anyone bother with the fc00::/7 addressing or do you use your public space (and police that)? kind regards Pshem On Fri, 9 Sep 2016 at 10:27 Mark Andrews wrote: > > In message SASAVsNX31Q_70Q+uDM1oeoHrQ at mail.gmail.com>, Pshem Kowalczyk writes: > > Hi, > > > > We're looking at rolling out IPv6 to our internal DC infrastructure. > Those > > systems support only our internal network and in the IPv4 world they all > > live in 'private' space of 10.0.0.0/8. I was wondering if anyone uses > the > > fc00::/7 space for these sort of things or do ppl use a bit of their > public > > IPv6 allocation and manage the security for those ranges? > > I realise I'd have to use a proxy or NAT66 for the regular outbound > > connectivity (but we do it already for IPv4 anyway). The truth is that > even > > if we do use something out of our public allocation we're likely to do > the > > same thing (just to be sure that nothing spills out accidentally). > > > > So what do you do in this space? > > > > kind regards > > Pshem > > If you have a NAT you can't prevent things spilling out. The ONLY > way to prevent things spilling out is to not connect the network > in any shape or form. > > All NAT does is make it harder to run your network and increases > the cost of software development. > > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org > From josh at kyneticwifi.com Thu Sep 8 23:15:01 2016 From: josh at kyneticwifi.com (Josh Reynolds) Date: Thu, 8 Sep 2016 18:15:01 -0500 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: You can also easily police a subnet. On Sep 8, 2016 6:11 PM, "Pshem Kowalczyk" wrote: > With NAT I have a single entry/exit point to those infrastructure subnets > which can be easily policed. > If I give them public IPs then they're routable and potentially can reach > the internet via devices that don't police the traffic. > > My real question is does anyone bother with the fc00::/7 addressing or do > you use your public space (and police that)? > > kind regards > Pshem > > > On Fri, 9 Sep 2016 at 10:27 Mark Andrews wrote: > > > > > In message > SASAVsNX31Q_70Q+uDM1oeoHrQ at mail.gmail.com>, Pshem Kowalczyk writes: > > > Hi, > > > > > > We're looking at rolling out IPv6 to our internal DC infrastructure. > > Those > > > systems support only our internal network and in the IPv4 world they > all > > > live in 'private' space of 10.0.0.0/8. I was wondering if anyone uses > > the > > > fc00::/7 space for these sort of things or do ppl use a bit of their > > public > > > IPv6 allocation and manage the security for those ranges? > > > I realise I'd have to use a proxy or NAT66 for the regular outbound > > > connectivity (but we do it already for IPv4 anyway). The truth is that > > even > > > if we do use something out of our public allocation we're likely to do > > the > > > same thing (just to be sure that nothing spills out accidentally). > > > > > > So what do you do in this space? > > > > > > kind regards > > > Pshem > > > > If you have a NAT you can't prevent things spilling out. The ONLY > > way to prevent things spilling out is to not connect the network > > in any shape or form. > > > > All NAT does is make it harder to run your network and increases > > the cost of software development. > > > > Mark > > -- > > Mark Andrews, ISC > > 1 Seymour St., Dundas Valley, NSW 2117, Australia > > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org > > > From Valdis.Kletnieks at vt.edu Thu Sep 8 23:21:19 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Thu, 08 Sep 2016 19:21:19 -0400 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: <33558.1473376879@turing-police.cc.vt.edu> On Thu, 08 Sep 2016 23:09:28 -0000, Pshem Kowalczyk said: > If I give them public IPs then they're routable and potentially can reach > the internet via devices that don't police the traffic. They can potentially reach the Internet even without public IPs. All it takes is one idiot with a laptop with a Wifi and Internet Connection Sharing. Or one miscreant. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 830 bytes Desc: not available URL: From marka at isc.org Thu Sep 8 23:27:46 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 09 Sep 2016 09:27:46 +1000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: Your message of "Thu, 08 Sep 2016 23:09:28 +0000." References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: <20160908232746.F140D5394DF2@rock.dv.isc.org> In message , Pshem Kowalczyk writes: > With NAT I have a single entry/exit point to those infrastructure subnets > which can be easily policed. > If I give them public IPs then they're routable and potentially can reach > the internet via devices that don't police the traffic. If you wish to believe that, believe that, but it is only wishful thinking. > My real question is does anyone bother with the fc00::/7 addressing or do > you use your public space (and police that)? ULA is normally used in parallel with public addressing if it is used. IPv6 was designed to be deployed with multiple address and prefixes per interface. When ULA is deployed you have ULA <-> ULA, non-ULA <-> non-ULA. Non-privacy addresses for server functionality, privacy addresses for client functionality. Mark > kind regards > Pshem -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From pshem.k at gmail.com Thu Sep 8 23:43:50 2016 From: pshem.k at gmail.com (Pshem Kowalczyk) Date: Thu, 08 Sep 2016 23:43:50 +0000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: <20160908232746.F140D5394DF2@rock.dv.isc.org> References: <20160908222656.62E5E53946A8@rock.dv.isc.org> <20160908232746.F140D5394DF2@rock.dv.isc.org> Message-ID: Hi, That's why I asked the question - if anyone actually puts its as an additional IP on their interfaces to keep it simple (and in-line with IPv4 policies, address allocation schemes, etc) or not. I can see the argument both ways - if we decide to use it we'll have to either overlay it with public IPv6 space (and provide the NAT/proxy for where we don't have any public IPv6 assigned) or simply not use the fc00::/7 and skip the NAT/proxy aspects of it. So one way it's aligned with what we do already (at the cost of the overhead) the other it's not aligned (but with potentially less overhead). kind regards Pshem On Fri, 9 Sep 2016 at 11:27 Mark Andrews wrote: > > In message < > CAEaZiRXU7DH9O9EwdjFiEMgDU7dt4v62W5+9+CTJ2-rqznP7Bg at mail.gmail.com>, > Pshem Kowalczyk writes: > > With NAT I have a single entry/exit point to those infrastructure subnets > > which can be easily policed. > > If I give them public IPs then they're routable and potentially can reach > > the internet via devices that don't police the traffic. > > If you wish to believe that, believe that, but it is only wishful > thinking. > > > My real question is does anyone bother with the fc00::/7 addressing > or do > you use your public space (and police that)? > > ULA is normally used in parallel with public addressing if it is > used. IPv6 was designed to be deployed with multiple address and > prefixes per interface. When ULA is deployed you have ULA <-> ULA, > non-ULA <-> non-ULA. Non-privacy addresses for server functionality, > privacy addresses for client functionality. > > Mark > > > kind regards > > Pshem > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org > From cb.list6 at gmail.com Fri Sep 9 00:17:23 2016 From: cb.list6 at gmail.com (Ca By) Date: Thu, 8 Sep 2016 17:17:23 -0700 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: On Thursday, September 8, 2016, Pshem Kowalczyk wrote: > With NAT I have a single entry/exit point to those infrastructure subnets > which can be easily policed. > If I give them public IPs then they're routable and potentially can reach > the internet via devices that don't police the traffic. > > My real question is does anyone bother with the fc00::/7 addressing or do Yes. That space is used for non-internet scenarios. NAT is bad CB > you use your public space (and police that)? > > kind regards > Pshem > > > On Fri, 9 Sep 2016 at 10:27 Mark Andrews > > wrote: > > > > > In message > SASAVsNX31Q_70Q+uDM1oeoHrQ at mail.gmail.com >, Pshem > Kowalczyk writes: > > > Hi, > > > > > > We're looking at rolling out IPv6 to our internal DC infrastructure. > > Those > > > systems support only our internal network and in the IPv4 world they > all > > > live in 'private' space of 10.0.0.0/8. I was wondering if anyone uses > > the > > > fc00::/7 space for these sort of things or do ppl use a bit of their > > public > > > IPv6 allocation and manage the security for those ranges? > > > I realise I'd have to use a proxy or NAT66 for the regular outbound > > > connectivity (but we do it already for IPv4 anyway). The truth is that > > even > > > if we do use something out of our public allocation we're likely to do > > the > > > same thing (just to be sure that nothing spills out accidentally). > > > > > > So what do you do in this space? > > > > > > kind regards > > > Pshem > > > > If you have a NAT you can't prevent things spilling out. The ONLY > > way to prevent things spilling out is to not connect the network > > in any shape or form. > > > > All NAT does is make it harder to run your network and increases > > the cost of software development. > > > > Mark > > -- > > Mark Andrews, ISC > > 1 Seymour St., Dundas Valley, NSW 2117, Australia > > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org > > > > From kauer at biplane.com.au Fri Sep 9 00:49:34 2016 From: kauer at biplane.com.au (Karl Auer) Date: Fri, 09 Sep 2016 10:49:34 +1000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> <20160908232746.F140D5394DF2@rock.dv.isc.org> Message-ID: <1473382174.4166.366.camel@biplane.com.au> On Thu, 2016-09-08 at 23:43 +0000, Pshem Kowalczyk wrote: > both ways - if we decide to use it we'll have to either overlay it > with public IPv6 space (and provide the NAT/proxy for where we don't > have any public IPv6 assigned) or simply not use the fc00::/7 and > skip the NAT/proxy aspects of it. There is no necessary link between ULA addresses and NAT. You don't have to NAT ULA, *ever*. If you need public addresses, go get them. There are enough. IMHO one should use ULA in only three situations: - a completely isolated network - for internal communications e.g. fabric management) - for testing Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer at biplane.com.au) http://www.biplane.com.au/kauer http://twitter.com/kauer389 GPG fingerprint: E00D 64ED 9C6A 8605 21E0 0ED0 EE64 2BEE CBCB C38B Old fingerprint: 3C41 82BE A9E7 99A1 B931 5AE7 7638 0147 2C3C 2AC4 From sryan at arbor.net Fri Sep 9 00:58:49 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Fri, 9 Sep 2016 00:58:49 +0000 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: <1473382174.4166.366.camel@biplane.com.au> References: <20160908222656.62E5E53946A8@rock.dv.isc.org> <20160908232746.F140D5394DF2@rock.dv.isc.org> , <1473382174.4166.366.camel@biplane.com.au> Message-ID: I agree with Karl. We use the ULA space for our internal test labs. The /48's we have in use get routed around internally but have no chance of leaking to the internet. Spencer Ryan | Senior Systems Administrator | sryan at arbor.net Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com ________________________________ From: NANOG on behalf of Karl Auer Sent: Thursday, September 8, 2016 8:49:34 PM To: nanog at nanog.org Subject: Re: Use of unique local IPv6 addressing rfc4193 On Thu, 2016-09-08 at 23:43 +0000, Pshem Kowalczyk wrote: > both ways - if we decide to use it we'll have to either overlay it > with public IPv6 space (and provide the NAT/proxy for where we don't > have any public IPv6 assigned) or simply not use the fc00::/7 and > skip the NAT/proxy aspects of it. There is no necessary link between ULA addresses and NAT. You don't have to NAT ULA, *ever*. If you need public addresses, go get them. There are enough. IMHO one should use ULA in only three situations: - a completely isolated network - for internal communications e.g. fabric management) - for testing Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer at biplane.com.au) http://www.biplane.com.au/kauer http://twitter.com/kauer389 GPG fingerprint: E00D 64ED 9C6A 8605 21E0 0ED0 EE64 2BEE CBCB C38B Old fingerprint: 3C41 82BE A9E7 99A1 B931 5AE7 7638 0147 2C3C 2AC4 From paula.wang at icann.org Thu Sep 8 18:24:34 2016 From: paula.wang at icann.org (Paula Wang) Date: Thu, 8 Sep 2016 18:24:34 +0000 Subject: IANA AS Numbers registry update Message-ID: <28F95FF5-4849-4559-BB9E-BC56A3B0DD07@icann.org> Hi, The IANA AS Numbers registry has been updated to reflect the allocation of the following block to LACNIC in September 2016: 265629-266652 Assigned by LACNIC 2016-09-08 You can find the IANA AS Numbers registry at: http://www.iana.org/assignments/as-numbers/as-numbers.xml The allocation was made in accordance with the Policy for Allocation of ASN Blocks to Regional Internet Registries: https://www.icann.org/resources/pages/global-policy-asn-blocks-2010-09-21-en Best Regards, Paula Wang IANA Specialist ICANN -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2046 bytes Desc: not available URL: From yang.yu.list at gmail.com Fri Sep 9 02:53:21 2016 From: yang.yu.list at gmail.com (Yang Yu) Date: Thu, 8 Sep 2016 21:53:21 -0500 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: On Thu, Sep 8, 2016 at 7:17 PM, Ca By wrote: > NAT is bad https://www.youtube.com/watch?v=v26BAlfWBm8 From octalnanog at alvarezp.org Fri Sep 9 08:02:37 2016 From: octalnanog at alvarezp.org (Octavio Alvarez) Date: Fri, 9 Sep 2016 01:02:37 -0700 Subject: Use of unique local IPv6 addressing rfc4193 In-Reply-To: References: <20160908222656.62E5E53946A8@rock.dv.isc.org> Message-ID: <57D26C9D.1030503@alvarezp.org> On 09/08/2016 04:09 PM, Pshem Kowalczyk wrote: > With NAT I have a single entry/exit point to those infrastructure subnets > which can be easily policed. I have used NAT in IPv4 scenarios as an alternative for lack of routing control in the return direction. However, this does not mean that this is the correct, best or orthodox way, even for IPv4, much less in IPv6. So, even though you can hack your way using NAT, this is really a routing problem, not an addressing problem. And you will just create problems for your users, your developers, yourself and third parties. > If I give them public IPs then they're routable and potentially can reach > the internet via devices that don't police the traffic. First: this can happen with NAT too. If other devices have access to the Internet, they can just NAT themselves even if the third-party exit has a private address. Second: private addresses can reach the Internet too. Many devices and ISPs don't block RFC5375-sourced packets from the Internet. So they can go out, although they can't come back. This is enough to create unsourceable attacks. In both cases NAT isn't really solving any of your problems fully. It's just a misconception. > My real question is does anyone bother with the fc00::/7 addressing or do > you use your public space (and police that)? I use public address space and I have made sure I have a single point of exit and return for my traffic. If I understood your concerns correctly, then I'd add that if the user forces traffic through third-party exit points, service is not guaranteed, as the third party may BCP38-filter it. If you want to absolutely prevent that, NAT will not help. You'll need other techniques. Best regards and good luck! From dcorbe at hammerfiber.com Fri Sep 9 15:39:16 2016 From: dcorbe at hammerfiber.com (Daniel Corbe) Date: Fri, 9 Sep 2016 11:39:16 -0400 Subject: Amazon Contact In-Reply-To: <2FC66B6EEF733844895E94A7FEE4FA5207373EC0@mbx032-e1-va-4.exch032.serverpod.net> References: <2FC66B6EEF733844895E94A7FEE4FA5207373EC0@mbx032-e1-va-4.exch032.serverpod.net> Message-ID: <1B104302-377E-4C94-9632-79A695D8AE2B@hammerfiber.com> > On Sep 8, 2016, at 1:54 PM, Shon Elliott wrote: > > Hi everyone, > > Sorry for having to ask this, but I haven't been able to chase down anyone from Amazon. Can someone from Amazon who might be watching the list who deals with the Amazon Instant Video, FireTV, Music, and other streaming media sections please contact me off-list regarding a serious issue? I would appreciate it very much. > > > Kind Regards, > Shon Elliott, KK6TOO > selliott at getunwired.com > > > I also need to chase down someone from Amazon. None of my customers seem to be able to access websites hosted on AWS. Including www.netflix.com and www.amazon.com proper. -Daniel From dougb at dougbarton.us Fri Sep 9 16:44:32 2016 From: dougb at dougbarton.us (Doug Barton) Date: Fri, 9 Sep 2016 09:44:32 -0700 Subject: =?UTF-8?Q?Israeli_Online_Attack_Service_=e2=80=98vDOS=e2=80=99_Earn?= =?UTF-8?Q?ed_$600=2c000_in_Two_Years?= Message-ID: vDOS ? a ?booter? service that has earned in excess of $600,000 over the past two years helping customers coordinate more than 150,000 so-called distributed denial-of-service (DDoS) attacks designed to knock Web sites offline ? has been massively hacked, spilling secrets about tens of thousands of paying customers and their targets. http://krebsonsecurity.com/2016/09/israeli-online-attack-service-vdos-earned-600000-in-two-years/ From cscora at apnic.net Fri Sep 9 18:01:36 2016 From: cscora at apnic.net (Routing Analysis Role Account) Date: Sat, 10 Sep 2016 04:01:36 +1000 (AEST) Subject: Weekly Routing Table Report Message-ID: <20160909180136.EC50CBF120@thyme.apnic.net> This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats at lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith . Routing Table Report 04:00 +10GMT Sat 10 Sep, 2016 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 608869 Prefixes after maximum aggregation (per Origin AS): 219887 Deaggregation factor: 2.77 Unique aggregates announced (without unneeded subnets): 297693 Total ASes present in the Internet Routing Table: 54797 Prefixes per ASN: 11.11 Origin-only ASes present in the Internet Routing Table: 36393 Origin ASes announcing only one prefix: 15419 Transit ASes present in the Internet Routing Table: 6519 Transit-only ASes present in the Internet Routing Table: 176 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 54 Max AS path prepend of ASN ( 55644) 51 Prefixes from unregistered ASNs in the Routing Table: 60 Unregistered ASNs in the Routing Table: 15 Number of 32-bit ASNs allocated by the RIRs: 15349 Number of 32-bit ASNs visible in the Routing Table: 11885 Prefixes from 32-bit ASNs in the Routing Table: 47401 Number of bogon 32-bit ASNs visible in the Routing Table: 68 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 402 Number of addresses announced to Internet: 2827227492 Equivalent to 168 /8s, 132 /16s and 17 /24s Percentage of available address space announced: 76.4 Percentage of allocated address space announced: 76.4 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.2 Total number of prefixes smaller than registry allocations: 197756 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 156029 Total APNIC prefixes after maximum aggregation: 42960 APNIC Deaggregation factor: 3.63 Prefixes being announced from the APNIC address blocks: 169453 Unique aggregates announced from the APNIC address blocks: 68913 APNIC Region origin ASes present in the Internet Routing Table: 5199 APNIC Prefixes per ASN: 32.59 APNIC Region origin ASes announcing only one prefix: 1164 APNIC Region transit ASes present in the Internet Routing Table: 947 Average APNIC Region AS path length visible: 4.4 Max APNIC Region AS path length visible: 54 Number of APNIC region 32-bit ASNs visible in the Routing Table: 2361 Number of APNIC addresses announced to Internet: 759442244 Equivalent to 45 /8s, 68 /16s and 43 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 183621 Total ARIN prefixes after maximum aggregation: 89506 ARIN Deaggregation factor: 2.05 Prefixes being announced from the ARIN address blocks: 189303 Unique aggregates announced from the ARIN address blocks: 88433 ARIN Region origin ASes present in the Internet Routing Table: 16204 ARIN Prefixes per ASN: 11.68 ARIN Region origin ASes announcing only one prefix: 5706 ARIN Region transit ASes present in the Internet Routing Table: 1713 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 1453 Number of ARIN addresses announced to Internet: 1107311840 Equivalent to 66 /8s, 0 /16s and 60 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 145625 Total RIPE prefixes after maximum aggregation: 71753 RIPE Deaggregation factor: 2.03 Prefixes being announced from the RIPE address blocks: 155882 Unique aggregates announced from the RIPE address blocks: 96418 RIPE Region origin ASes present in the Internet Routing Table: 18119 RIPE Prefixes per ASN: 8.60 RIPE Region origin ASes announcing only one prefix: 7832 RIPE Region transit ASes present in the Internet Routing Table: 3028 Average RIPE Region AS path length visible: 4.4 Max RIPE Region AS path length visible: 27 Number of RIPE region 32-bit ASNs visible in the Routing Table: 5039 Number of RIPE addresses announced to Internet: 708201216 Equivalent to 42 /8s, 54 /16s and 75 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 61547 Total LACNIC prefixes after maximum aggregation: 12383 LACNIC Deaggregation factor: 4.97 Prefixes being announced from the LACNIC address blocks: 77216 Unique aggregates announced from the LACNIC address blocks: 37336 LACNIC Region origin ASes present in the Internet Routing Table: 2474 LACNIC Prefixes per ASN: 31.21 LACNIC Region origin ASes announcing only one prefix: 543 LACNIC Region transit ASes present in the Internet Routing Table: 584 Average LACNIC Region AS path length visible: 4.7 Max LACNIC Region AS path length visible: 22 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 2771 Number of LACNIC addresses announced to Internet: 170757184 Equivalent to 10 /8s, 45 /16s and 140 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-266652 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 14585 Total AfriNIC prefixes after maximum aggregation: 3275 AfriNIC Deaggregation factor: 4.45 Prefixes being announced from the AfriNIC address blocks: 16613 Unique aggregates announced from the AfriNIC address blocks: 6268 AfriNIC Region origin ASes present in the Internet Routing Table: 733 AfriNIC Prefixes per ASN: 22.66 AfriNIC Region origin ASes announcing only one prefix: 174 AfriNIC Region transit ASes present in the Internet Routing Table: 180 Average AfriNIC Region AS path length visible: 4.6 Max AfriNIC Region AS path length visible: 18 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 261 Number of AfriNIC addresses announced to Internet: 81113088 Equivalent to 4 /8s, 213 /16s and 176 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5547 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3531 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 3116 11143 1066 KIXS-AS-KR Korea Telecom, KR 17974 2942 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2683 1496 528 BSNL-NIB National Internet Backbone, IN 9808 2251 8781 42 CMNET-GD Guangdong Mobile Communication 4755 2051 429 222 TATACOMM-AS TATA Communications formerl 4808 1760 2295 528 CHINA169-BJ China Unicom Beijing Provin 45899 1574 1231 90 VNPT-AS-VN VNPT Corp, VN 24560 1538 508 220 AIRTELBROADBAND-AS-AP Bharti Airtel Ltd Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 22773 3512 2964 146 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 6389 2219 3671 41 BELLSOUTH-NET-BLK - BellSouth.net Inc., 18566 2195 405 110 MEGAPATH5-US - MegaPath Corporation, US 20115 1937 1977 403 CHARTER-NET-HKY-NC - Charter Communicat 30036 1725 337 345 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 209 1702 5069 649 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 6983 1677 848 227 ITCDELTA - Earthlink, Inc., US 16509 1395 2532 458 AMAZON-02 - Amazon.com, Inc., US 7018 1369 20093 1007 ATT-INTERNET4 - AT&T Services, Inc., US 11492 1285 229 622 CABLEONE - CABLE ONE, INC., US Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3329 169 15 ALJAWWALSTC-AS , SA 20940 2819 1077 1997 AKAMAI-ASN1 , US 34984 1974 327 356 TELLCOM-AS , TR 12479 1329 1018 46 UNI2-AS , ES 13188 1223 99 65 BANKINFORM-AS , UA 8551 1205 377 46 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 6849 1144 355 21 UKRTELNET , UA 8402 1074 544 15 CORBINA-AS Russia, RU 9198 936 352 25 KAZTELECOM-AS , KZ 12389 923 1091 400 ROSTELECOM-AS , RU Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 10620 3479 541 167 Telmex Colombia S.A., CO 8151 2287 3417 568 Uninet S.A. de C.V., MX 7303 1544 954 245 Telecom Argentina S.A., AR 6503 1396 437 55 Axtel, S.A.B. de C.V., MX 11830 1325 367 57 Instituto Costarricense de Electricidad 6147 1081 377 27 Telefonica del Peru S.A.A., PE 7738 994 1882 40 Telemar Norte Leste S.A., BR 3816 962 464 209 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 11172 907 125 76 Alestra, S. de R.L. de C.V., MX 28573 897 2180 162 CLARO S.A., BR Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1179 402 50 LINKdotNET-AS, EG 36903 684 344 115 MT-MPLS, MA 37611 647 48 2 Afrihost, ZA 36992 547 1373 25 ETISALAT-MISR, EG 8452 540 1472 15 TE-AS TE-AS, EG 37492 384 250 69 ORANGE-TN, TN 24835 345 610 16 RAYA-AS, EG 29571 301 37 12 CITelecom-AS, CI 15399 296 35 6 WANANCHI-KE, KE 2018 264 327 74 TENET-1, ZA Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5547 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3531 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 22773 3512 2964 146 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3479 541 167 Telmex Colombia S.A., CO 39891 3329 169 15 ALJAWWALSTC-AS , SA 4766 3116 11143 1066 KIXS-AS-KR Korea Telecom, KR 17974 2942 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 20940 2819 1077 1997 AKAMAI-ASN1 , US 9829 2683 1496 528 BSNL-NIB National Internet Backbone, IN 8151 2287 3417 568 Uninet S.A. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 22773 3512 3366 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 39891 3329 3314 ALJAWWALSTC-AS , SA 10620 3479 3312 Telmex Colombia S.A., CO 7545 3531 3281 TPG-INTERNET-AP TPG Telecom Limited, AU 17974 2942 2864 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9808 2251 2209 CMNET-GD Guangdong Mobile Communication 6389 2219 2178 BELLSOUTH-NET-BLK - BellSouth.net Inc., 9829 2683 2155 BSNL-NIB National Internet Backbone, IN 18566 2195 2085 MEGAPATH5-US - MegaPath Corporation, US 4766 3116 2050 KIXS-AS-KR Korea Telecom, KR Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65001 PRIVATE 5.143.176.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.200.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.208.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.216.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65000 PRIVATE 31.219.177.0/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65000 PRIVATE 31.219.177.128/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65512 PRIVATE 45.252.244.0/24 45899 VNPT-AS-VN VNPT Corp, VN 65512 PRIVATE 45.252.245.0/24 45899 VNPT-AS-VN VNPT Corp, VN Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 23.249.144.0/20 40430 COLO4JAX-AS - colo4jax, LLC, US 24.49.144.0/20 30164 USACOMMUNICATIONS-1 - USA COMPANIES, LP 27.100.7.0/24 56096 UNKNOWN 41.73.1.0/24 37004 -Reserved AS-, ZZ 41.73.2.0/24 37004 -Reserved AS-, ZZ 41.73.3.0/24 37004 -Reserved AS-, ZZ 41.73.4.0/24 37004 -Reserved AS-, ZZ 41.73.5.0/24 37004 -Reserved AS-, ZZ 41.73.6.0/24 37004 -Reserved AS-, ZZ 41.73.7.0/24 37004 -Reserved AS-, ZZ Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:16 /9:13 /10:36 /11:101 /12:269 /13:518 /14:1052 /15:1767 /16:13140 /17:7822 /18:12982 /19:25320 /20:38642 /21:40272 /22:67769 /23:59371 /24:338263 /25:531 /26:510 /27:344 /28:52 /29:32 /30:13 /31:1 /32:33 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 39891 2896 3329 ALJAWWALSTC-AS , SA 22773 2737 3512 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2087 2195 MEGAPATH5-US - MegaPath Corporation, US 30036 1539 1725 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 6389 1418 2219 BELLSOUTH-NET-BLK - BellSouth.net Inc., 10620 1396 3479 Telmex Colombia S.A., CO 6983 1327 1677 ITCDELTA - Earthlink, Inc., US 34984 1257 1974 TELLCOM-AS , TR 11492 1187 1285 CABLEONE - CABLE ONE, INC., US 13188 999 1223 BANKINFORM-AS , UA Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1619 2:770 4:21 5:2178 6:31 8:984 12:1803 13:42 14:1755 15:45 16:2 17:95 18:126 20:50 22:1 23:1705 24:1797 27:2322 31:1787 32:85 33:4 35:5 36:313 37:2310 38:1252 39:36 40:96 41:2885 42:454 43:1852 44:47 45:2142 46:2575 47:93 49:1214 50:900 51:13 52:546 54:345 55:7 56:7 57:42 58:1564 59:1006 60:381 61:1813 62:1513 63:1909 64:4541 65:2185 66:4248 67:2213 68:1116 69:3254 70:1280 71:525 72:2046 74:2544 75:398 76:415 77:1374 78:1225 79:910 80:1305 81:1400 82:983 83:715 84:853 85:1570 86:486 87:1123 88:552 89:2077 90:202 91:6060 92:945 93:2364 94:2415 95:2478 96:519 97:340 98:947 99:90 100:78 101:1158 103:12243 104:2489 105:126 106:437 107:1391 108:660 109:2259 110:1276 111:1629 112:1050 113:1143 114:1042 115:1680 116:1649 117:1542 118:2038 119:1578 120:930 121:1118 122:2263 123:2015 124:1560 125:1832 128:666 129:457 130:412 131:1383 132:611 133:176 134:484 135:145 136:381 137:409 138:1891 139:422 140:577 141:454 142:662 143:981 144:737 145:162 146:945 147:655 148:1390 149:529 150:664 151:878 152:653 153:297 154:683 155:947 156:528 157:505 158:389 159:1228 160:511 161:732 162:2381 163:535 164:772 165:1101 166:333 167:1195 168:2140 169:666 170:1908 171:260 172:684 173:1688 174:758 175:670 176:1736 177:4005 178:2240 179:1157 180:2158 181:1860 182:1998 183:996 184:824 185:7343 186:3166 187:2111 188:2171 189:1757 190:7638 191:1273 192:9033 193:5721 194:4459 195:3850 196:1761 197:1136 198:5563 199:5756 200:7120 201:3660 202:10105 203:9830 204:4511 205:2739 206:2960 207:3087 208:4020 209:3860 210:3888 211:2037 212:2744 213:2331 214:838 215:70 216:5803 217:1987 218:797 219:609 220:1633 221:884 222:696 223:1281 End of report From dwhite at olp.net Fri Sep 9 19:52:40 2016 From: dwhite at olp.net (Dan White) Date: Fri, 9 Sep 2016 14:52:40 -0500 Subject: Looking for recommendations for a dedicated ping responder Message-ID: <20160909195239.GF3870@dan.olp.net> Are there any products you're using which are dedicated to responding to customer facing pings? -- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite at olp.net http://www.btcbroadband.com From laszlo at heliacal.net Fri Sep 9 19:58:20 2016 From: laszlo at heliacal.net (Laszlo Hanyecz) Date: Fri, 9 Sep 2016 19:58:20 +0000 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: <20160909195239.GF3870@dan.olp.net> References: <20160909195239.GF3870@dan.olp.net> Message-ID: <59de5345-94d3-3139-de10-38851dc7a510@heliacal.net> On 2016-09-09 19:52, Dan White wrote: > Are there any products you're using which are dedicated to responding to > customer facing pings? > PaaS (pong-as-a-service)? From josh at kyneticwifi.com Fri Sep 9 20:00:34 2016 From: josh at kyneticwifi.com (Josh Reynolds) Date: Fri, 9 Sep 2016 15:00:34 -0500 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: <20160909195239.GF3870@dan.olp.net> References: <20160909195239.GF3870@dan.olp.net> Message-ID: Can you elaborate? On Sep 9, 2016 2:54 PM, "Dan White" wrote: > Are there any products you're using which are dedicated to responding to > customer facing pings? > > -- > Dan White > BTC Broadband > Network Admin Lead > Ph 918.366.0248 (direct) main: (918)366-8000 > Fax 918.366.6610 email: dwhite at olp.net > http://www.btcbroadband.com > From dwhite at olp.net Fri Sep 9 20:08:49 2016 From: dwhite at olp.net (Dan White) Date: Fri, 9 Sep 2016 15:08:49 -0500 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: References: <20160909195239.GF3870@dan.olp.net> Message-ID: <20160909200848.GH3870@dan.olp.net> We're being caught up in some sort of peering dispute between Level 3 and Google (in the Dallas area), and we've fielded several calls from larger customers complaining of 40-50% packet loss (to 8.8.8.8) when there appears to be no actual service impacting loss. We currently suggest customers use a Linux server to ping against, or another public host. Ideally we'd like to use a hardware based ICMP system for customer use - Accedian NIDs are good at this (exceptionally low jitter) accept they throttle at 500 pings per second. On 09/09/16?15:00?-0500, Josh Reynolds wrote: >Can you elaborate? > >On Sep 9, 2016 2:54 PM, "Dan White" wrote: > >> Are there any products you're using which are dedicated to responding to >> customer facing pings? -- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite at olp.net http://www.btcbroadband.com From jared at puck.nether.net Fri Sep 9 20:17:33 2016 From: jared at puck.nether.net (Jared Mauch) Date: Fri, 9 Sep 2016 16:17:33 -0400 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: <20160909200848.GH3870@dan.olp.net> References: <20160909195239.GF3870@dan.olp.net> <20160909200848.GH3870@dan.olp.net> Message-ID: > On Sep 9, 2016, at 4:08 PM, Dan White wrote: > > We're being caught up in some sort of peering dispute between Level 3 and > Google (in the Dallas area), and we've fielded several calls from larger > customers complaining of 40-50% packet loss (to 8.8.8.8) when there appears > to be no actual service impacting loss. > > We currently suggest customers use a Linux server to ping against, or > another public host. > > Ideally we'd like to use a hardware based ICMP system for customer use - > Accedian NIDs are good at this (exceptionally low jitter) accept they > throttle at 500 pings per second. I know that the NETNOD folks did NTP in a FPGA that can do 4x 10GE, perhaps that card and code could be used to do 40G ICMP responder? - Jared From morrowc.lists at gmail.com Fri Sep 9 20:42:51 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Fri, 9 Sep 2016 16:42:51 -0400 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: References: <20160909195239.GF3870@dan.olp.net> <20160909200848.GH3870@dan.olp.net> Message-ID: On Fri, Sep 9, 2016 at 4:17 PM, Jared Mauch wrote: > > > On Sep 9, 2016, at 4:08 PM, Dan White wrote: > > > > We're being caught up in some sort of peering dispute between Level 3 and > > Google (in the Dallas area), and we've fielded several calls from larger > > customers complaining of 40-50% packet loss (to 8.8.8.8) when there > appears > > to be no actual service impacting loss. > > > > We currently suggest customers use a Linux server to ping against, or > > another public host. > > > > Ideally we'd like to use a hardware based ICMP system for customer use - > > Accedian NIDs are good at this (exceptionally low jitter) accept they > > throttle at 500 pings per second. > > I know that the NETNOD folks did NTP in a FPGA that can do 4x 10GE, > perhaps that card and code could be used to do 40G ICMP responder? > > or, alternately test some useful application instead? I mean, 'wget' will tell you stats about the bw/etc... apache-bench will as well, and you can probably whip up some custom python/etc that'd do the same sort of thing. From jlewis at lewis.org Fri Sep 9 22:29:28 2016 From: jlewis at lewis.org (Jon Lewis) Date: Fri, 9 Sep 2016 18:29:28 -0400 (EDT) Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: References: <20160909195239.GF3870@dan.olp.net> <20160909200848.GH3870@dan.olp.net> Message-ID: On Fri, 9 Sep 2016, Jared Mauch wrote: > >> On Sep 9, 2016, at 4:08 PM, Dan White wrote: >> >> We're being caught up in some sort of peering dispute between Level 3 and >> Google (in the Dallas area), and we've fielded several calls from larger >> customers complaining of 40-50% packet loss (to 8.8.8.8) when there appears >> to be no actual service impacting loss. >> >> We currently suggest customers use a Linux server to ping against, or >> another public host. >> >> Ideally we'd like to use a hardware based ICMP system for customer use - >> Accedian NIDs are good at this (exceptionally low jitter) accept they >> throttle at 500 pings per second. > > I know that the NETNOD folks did NTP in a FPGA that can do 4x 10GE, > perhaps that card and code could be used to do 40G ICMP responder? The trouble is, LOTS of people want to ping something "out on the internet" to verify their connectivity, and things like GOOG's 8.8.8.8 DNS servers are a popular lighthouse. I know from first hand experience (dealing with customers complaining about it), that GOOG, at least at some of the anycast nodes for the service, polices ICMP echo requests aimed at 8.8.8.8 due to the quantity of those unwanted packets. Having a cheap/small/powerful device that can be used as a ping target, and getting the masses to use it are two very different things. Dan, are your customers missing DNS responses, or just echo replies from 8.8.8.8? If the latter, ask what they'd do if thousands of people pinged one of their servers constantly. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ From james at mor-pah.net Sat Sep 10 08:57:50 2016 From: james at mor-pah.net (James Greig) Date: Sat, 10 Sep 2016 09:57:50 +0100 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: References: <20160909195239.GF3870@dan.olp.net> <20160909200848.GH3870@dan.olp.net> Message-ID: <7AF8E07A-2AD6-40D9-AFEC-A61FD6BF50F4@mor-pah.net> On one of these lists around 6 months ago a Google network engineer confirmed they do rate limit icmp (aside from prioritisation). Unless there's a real issue here this is more about educating people. It's amazing how many still miss interpret trace routes these days. Kind regards James Greig > On 9 Sep 2016, at 23:29, Jon Lewis wrote: > >> On Fri, 9 Sep 2016, Jared Mauch wrote: >> >> >>> On Sep 9, 2016, at 4:08 PM, Dan White wrote: >>> >>> We're being caught up in some sort of peering dispute between Level 3 and >>> Google (in the Dallas area), and we've fielded several calls from larger >>> customers complaining of 40-50% packet loss (to 8.8.8.8) when there appears >>> to be no actual service impacting loss. >>> >>> We currently suggest customers use a Linux server to ping against, or >>> another public host. >>> >>> Ideally we'd like to use a hardware based ICMP system for customer use - >>> Accedian NIDs are good at this (exceptionally low jitter) accept they >>> throttle at 500 pings per second. >> >> I know that the NETNOD folks did NTP in a FPGA that can do 4x 10GE, >> perhaps that card and code could be used to do 40G ICMP responder? > > The trouble is, LOTS of people want to ping something "out on the internet" to verify their connectivity, and things like GOOG's 8.8.8.8 DNS servers are a popular lighthouse. I know from first hand experience (dealing with customers complaining about it), that GOOG, at least at some of the anycast nodes for the service, polices ICMP echo requests aimed at > 8.8.8.8 due to the quantity of those unwanted packets. > > Having a cheap/small/powerful device that can be used as a ping target, and getting the masses to use it are two very different things. > > Dan, are your customers missing DNS responses, or just echo replies from 8.8.8.8? If the latter, ask what they'd do if thousands of people pinged one of their servers constantly. > > ---------------------------------------------------------------------- > Jon Lewis, MCP :) | I route > | therefore you are > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ From list at satchell.net Sat Sep 10 13:55:59 2016 From: list at satchell.net (Stephen Satchell) Date: Sat, 10 Sep 2016 06:55:59 -0700 Subject: Status of IPv6 on Charter Communications Message-ID: Would someone at Charter Communications who is on this list indicate the roll-out schedule for IPv6 to business customers using cable modems as opposed to fiber links? From matthew at matthew.at Sat Sep 10 16:21:02 2016 From: matthew at matthew.at (Matthew Kaufman) Date: Sat, 10 Sep 2016 16:21:02 +0000 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: <7AF8E07A-2AD6-40D9-AFEC-A61FD6BF50F4@mor-pah.net> References: <20160909195239.GF3870@dan.olp.net> <20160909200848.GH3870@dan.olp.net> <7AF8E07A-2AD6-40D9-AFEC-A61FD6BF50F4@mor-pah.net> Message-ID: Personally, I'd think twice before putting a box that does unthrottled reflection of ICMP packets to their claimed source anywhere, especially not one with a well-known address. Matthew Kaufman On Sat, Sep 10, 2016 at 2:01 AM James Greig wrote: > On one of these lists around 6 months ago a Google network engineer > confirmed they do rate limit icmp (aside from prioritisation). > > Unless there's a real issue here this is more about educating people. > It's amazing how many still miss interpret trace routes these days. > > Kind regards > > James Greig > > > On 9 Sep 2016, at 23:29, Jon Lewis wrote: > > > >> On Fri, 9 Sep 2016, Jared Mauch wrote: > >> > >> > >>> On Sep 9, 2016, at 4:08 PM, Dan White wrote: > >>> > >>> We're being caught up in some sort of peering dispute between Level 3 > and > >>> Google (in the Dallas area), and we've fielded several calls from > larger > >>> customers complaining of 40-50% packet loss (to 8.8.8.8) when there > appears > >>> to be no actual service impacting loss. > >>> > >>> We currently suggest customers use a Linux server to ping against, or > >>> another public host. > >>> > >>> Ideally we'd like to use a hardware based ICMP system for customer use > - > >>> Accedian NIDs are good at this (exceptionally low jitter) accept they > >>> throttle at 500 pings per second. > >> > >> I know that the NETNOD folks did NTP in a FPGA that can do 4x 10GE, > >> perhaps that card and code could be used to do 40G ICMP responder? > > > > The trouble is, LOTS of people want to ping something "out on the > internet" to verify their connectivity, and things like GOOG's 8.8.8.8 DNS > servers are a popular lighthouse. I know from first hand experience > (dealing with customers complaining about it), that GOOG, at least at some > of the anycast nodes for the service, polices ICMP echo requests aimed at > > 8.8.8.8 due to the quantity of those unwanted packets. > > > > Having a cheap/small/powerful device that can be used as a ping target, > and getting the masses to use it are two very different things. > > > > Dan, are your customers missing DNS responses, or just echo replies from > 8.8.8.8? If the latter, ask what they'd do if thousands of people pinged > one of their servers constantly. > > > > ---------------------------------------------------------------------- > > Jon Lewis, MCP :) | I route > > | therefore you are > > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ > > From randy at psg.com Sun Sep 11 18:34:43 2016 From: randy at psg.com (Randy Bush) Date: Sun, 11 Sep 2016 11:34:43 -0700 Subject: comcast and msoft ports Message-ID: anyone know if comcast residential filters 139/445? randy From cb.list6 at gmail.com Sun Sep 11 18:38:49 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 11 Sep 2016 11:38:49 -0700 Subject: comcast and msoft ports In-Reply-To: References: Message-ID: On Sunday, September 11, 2016, Randy Bush wrote: > anyone know if comcast residential filters 139/445? > > randy > https://customer.xfinity.com/help-and-support/internet/list-of-blocked-ports/ From sryan at arbor.net Sun Sep 11 18:39:56 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Sun, 11 Sep 2016 18:39:56 +0000 Subject: comcast and msoft ports In-Reply-To: References: Message-ID: <3v2i4pdfn2u0docv87ho1ad5.1473619194234@email.android.com> https://customer.xfinity.com/help-and-support/internet/list-of-blocked-ports/ Sent from my Verizon, Samsung Galaxy smartphone -------- Original message -------- From: Randy Bush Date: 9/11/16 2:35 PM (GMT-05:00) To: North American Network Operators' Group Subject: comcast and msoft ports anyone know if comcast residential filters 139/445? randy From randy at psg.com Sun Sep 11 18:42:58 2016 From: randy at psg.com (Randy Bush) Date: Sun, 11 Sep 2016 11:42:58 -0700 Subject: comcast and msoft ports In-Reply-To: References: Message-ID: sigh. well that was some fun hours debugging; not. thanks randy From sryan at arbor.net Sun Sep 11 19:21:10 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Sun, 11 Sep 2016 19:21:10 +0000 Subject: comcast and msoft ports In-Reply-To: References: , Message-ID: Having those ports exposed to the Internet is scary. Comcast is right in blocking them. Sent from my Verizon, Samsung Galaxy smartphone -------- Original message -------- From: Randy Bush Date: 9/11/16 2:48 PM (GMT-05:00) To: Ca By Cc: North American Network Operators' Group Subject: Re: comcast and msoft ports sigh. well that was some fun hours debugging; not. thanks randy From fhr at fhrnet.eu Sun Sep 11 19:25:04 2016 From: fhr at fhrnet.eu (Filip Hruska) Date: Sun, 11 Sep 2016 21:25:04 +0200 Subject: comcast and msoft ports In-Reply-To: References: Message-ID: <4f0973eb-9bd8-6719-fc8e-a0724e62dd94@fhrnet.eu> If you really need them, you'll need to use some sort of tunneling mechanism, ie PPTP. Regards, Filip On 11.9.2016 21:21, Ryan, Spencer wrote: > Having those ports exposed to the Internet is scary. Comcast is right in blocking them. > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > -------- Original message -------- > From: Randy Bush > Date: 9/11/16 2:48 PM (GMT-05:00) > To: Ca By > Cc: North American Network Operators' Group > Subject: Re: comcast and msoft ports > > sigh. well that was some fun hours debugging; not. > > thanks > > randy > From swmike at swm.pp.se Sun Sep 11 19:30:04 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Sun, 11 Sep 2016 21:30:04 +0200 (CEST) Subject: comcast and msoft ports In-Reply-To: References: Message-ID: On Sun, 11 Sep 2016, Randy Bush wrote: > sigh. well that was some fun hours debugging; not. 135/137/139/445 has seen widespread filtering since... errr.. 2000? I know it was widely done back in those days when people were connecting their computers directly to the bridged modem/ETTH jack and things ended up in the newspapers that peoples files were available on the Internet because they didn't set a password on their windows share.... or when versions of Windows were pwned during installation of Windows because... Windows. -- Mikael Abrahamsson email: swmike at swm.pp.se From cb.list6 at gmail.com Sun Sep 11 20:02:17 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 11 Sep 2016 13:02:17 -0700 Subject: comcast and msoft ports In-Reply-To: <4f0973eb-9bd8-6719-fc8e-a0724e62dd94@fhrnet.eu> References: <4f0973eb-9bd8-6719-fc8e-a0724e62dd94@fhrnet.eu> Message-ID: On Sunday, September 11, 2016, Filip Hruska wrote: > If you really need them, you'll need to use some sort of tunneling > mechanism, ie PPTP. > > Friendly reminder, next week ios 10 drops Prepare servers for iOS 10 & macOS Sierra. Crypto Deprecations: - SSLv3 - RC4 - PPTP VPN support.apple.com/en-us/HT206871 support.apple.com/en-us/HT206844 Regards, > Filip > > On 11.9.2016 21:21, Ryan, Spencer wrote: > >> Having those ports exposed to the Internet is scary. Comcast is right in >> blocking them. >> >> >> >> Sent from my Verizon, Samsung Galaxy smartphone >> >> >> -------- Original message -------- >> From: Randy Bush >> Date: 9/11/16 2:48 PM (GMT-05:00) >> To: Ca By >> Cc: North American Network Operators' Group >> Subject: Re: comcast and msoft ports >> >> sigh. well that was some fun hours debugging; not. >> >> thanks >> >> randy >> >> From dhill+nanog at mindcry.org Sat Sep 10 15:14:13 2016 From: dhill+nanog at mindcry.org (David Hill) Date: Sat, 10 Sep 2016 11:14:13 -0400 Subject: Status of IPv6 on Charter Communications In-Reply-To: References: Message-ID: <20160910151413.so36rwjawtuhlvai@560338596deac36a23b4268bcf651d8275290d1da332da34> On Sat, Sep 10, 2016 at 06:55:59AM -0700, Stephen Satchell wrote: > Would someone at Charter Communications who is on this list indicate the > roll-out schedule for IPv6 to business customers using cable modems as > opposed to fiber links? I too would appreciate this information. I do see more networks being announced: http://bgp.he.net/AS20115#_prefixes6. Hopefully they will be providing static IPv6 as well. - David From hugo at slabnet.com Sun Sep 11 20:54:18 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Sun, 11 Sep 2016 13:54:18 -0700 Subject: "Defensive" BGP hijacking? Message-ID: Hopefully this is operational enough, though obviously leaning more towards the policy side of things: What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? -- Hugo Slabbert?????? | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E?? | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 850 bytes Desc: not available URL: From fhr at fhrnet.eu Sun Sep 11 21:36:19 2016 From: fhr at fhrnet.eu (FHR) Date: Sun, 11 Sep 2016 23:36:19 +0200 Subject: "Defensive" BGP hijacking? In-Reply-To: Message-ID: <0c7701eb-a0e7-476b-b0f3-6f7fa58cbd04@email.android.com> From cb.list6 at gmail.com Sun Sep 11 23:51:38 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 11 Sep 2016 16:51:38 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: On Sunday, September 11, 2016, Hugo Slabbert wrote: > Hopefully this is operational enough, though obviously leaning more > towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for > defensive purposes"? Not ok. Never. > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in- > israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting > us,? Townsend explained. ?What we were doing was for defensive purposes. We > were simply trying to get them to stop and to gather as much information as > possible about the botnet they were using and report that to the proper > authorities.? > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal From frnkblk at iname.com Mon Sep 12 03:29:58 2016 From: frnkblk at iname.com (frnkblk at iname.com) Date: Sun, 11 Sep 2016 22:29:58 -0500 Subject: Optical transceiver question In-Reply-To: References: <001001d20945$b2dc9680$1895c380$@iname.com> Message-ID: <005801d20ca5$f1431c10$d3c95430$@iname.com> In discussions with the reseller he admitted that they market the distance based on average TX power and average link loss, so it is possible to purchase optics that may not be able to attain certain necessary link budgets and therefore distances. There are 1270/1310 nm BiDi optics with a worst-case link margin of 19 dB. Assuming a loss of 0.38 dB/km (https://www.cisco.com/c/en/us/support/docs/optical-networking/ons-15454-son et-multiservice-provisioning-platform-mspp/27042-max-att-27042.html) that's just 50 km. Of course, with a link margin of 24 dB that would be 61.5 km. So unless you assume best-cast scenarios, 60 km is a stretch. Frank -----Original Message----- From: Mikael Abrahamsson [mailto:swmike at swm.pp.se] Sent: Thursday, September 08, 2016 1:36 PM To: Frank Bulk Cc: nanog at nanog.org Subject: Re: Optical transceiver question On Wed, 7 Sep 2016, Frank Bulk wrote: > Is it an industry practice to market distance based on the hot optics, > not on the worst case, which is minimum TX power? No. If this is 1310nm optics with 0.4dB/km budget, the budget figure should be end-of-life figure, ie worst case according to the specs. I don't like the "kilometer" figures, that can be marketed with very optimistic figures. However, if the transceiver says 0 to -5 transmit, if it doesn't transmit 0 to -5 then it's out of spec. I treat the kilometer figure as "marketing", and look only at the optical specifications. So using your figures, if the device doesn't have 0 to -5 out, and can receive error free at -20, then it's out of spec and it should be replaced free of charge. However, if they market 1310nm with 15dB link budget at 60km reach, then I'd consder that false marketing. -- Mikael Abrahamsson email: swmike at swm.pp.se From jared at puck.nether.net Mon Sep 12 11:43:45 2016 From: jared at puck.nether.net (Jared Mauch) Date: Mon, 12 Sep 2016 07:43:45 -0400 Subject: comcast and msoft ports In-Reply-To: References: <4f0973eb-9bd8-6719-fc8e-a0724e62dd94@fhrnet.eu> Message-ID: > On Sep 11, 2016, at 4:02 PM, Ca By wrote: > > On Sunday, September 11, 2016, Filip Hruska wrote: > >> If you really need them, you'll need to use some sort of tunneling >> mechanism, ie PPTP. >> >> > > Friendly reminder, next week ios 10 drops > > > Prepare servers for iOS 10 & macOS Sierra. Crypto Deprecations: > - SSLv3 > - RC4 > - PPTP VPN > support.apple.com/en-us/HT206871 > support.apple.com/en-us/HT206844 > And expect your SSH DSA keys to require a workaround, or just generate new ecdsa and RSA keys. - Jared From GHeimer at mc3.edu Sun Sep 11 21:04:37 2016 From: GHeimer at mc3.edu (Gregg Heimer) Date: Sun, 11 Sep 2016 21:04:37 +0000 Subject: comcast and msoft ports Message-ID: Yes of course they do. If you need NetBIOS and SMB, create a VPN tunnel. List of ports https://customer.xfinity.com/help-and-support/internet/list-of-blocked-ports/ On Sep 11, 2016 2:45 PM, Ca By wrote: On Sunday, September 11, 2016, Randy Bush wrote: > anyone know if comcast residential filters 139/445? > > randy > https://customer.xfinity.com/help-and-support/internet/list-of-blocked-ports/ ________________________________ Montgomery County Community College is proud to be designated as an Achieving the Dream Leader College for its commitment to student access and success. From jared at puck.nether.net Mon Sep 12 12:46:30 2016 From: jared at puck.nether.net (Jared Mauch) Date: Mon, 12 Sep 2016 08:46:30 -0400 Subject: comcast and msoft ports In-Reply-To: References: <4f0973eb-9bd8-6719-fc8e-a0724e62dd94@fhrnet.eu> Message-ID: > On Sep 12, 2016, at 7:43 AM, jared mauch wrote: > > And expect your SSH DSA keys to require a workaround, or just generate new ecdsa and RSA keys. Sorry, brain-keyboard output meant to say: ED25519 [-t dsa | ecdsa | ed25519 | rsa | rsa1] https://www.gentoo.org/support/news-items/2015-08-13-openssh-weak-keys.html macOS sierra inherits this as it provides OpenSSH 7.2. - jared From blake at ispn.net Mon Sep 12 15:24:03 2016 From: blake at ispn.net (Blake Hudson) Date: Mon, 12 Sep 2016 10:24:03 -0500 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net> Hugo Slabbert wrote on 9/11/2016 3:54 PM: > Hopefully this is operational enough, though obviously leaning more towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? > https://bgpstream.com/event/54711 My suggestion is that BackConnect/Bryant Townsend should have their ASN revoked for fraudulently announcing another organization's address space. They are not law enforcement, they did not have a warrant or judicial oversight, they were not in immediate mortal peril, etc, etc. From sryan at arbor.net Mon Sep 12 15:47:16 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Mon, 12 Sep 2016 15:47:16 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net> References: , <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net> Message-ID: I'm in the "never acceptable" camp. Filtering routes/peers? Sure. Disconnecting one of your own customers to stop an attack originating from them? Sure. Hijacking an AS you have no permission to control? No. Obviously my views and not of my employer. Spencer Ryan | Senior Systems Administrator | sryan at arbor.net Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com ________________________________ From: NANOG on behalf of Blake Hudson Sent: Monday, September 12, 2016 11:24:03 AM To: nanog at nanog.org Subject: Re: "Defensive" BGP hijacking? Hugo Slabbert wrote on 9/11/2016 3:54 PM: > Hopefully this is operational enough, though obviously leaning more towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? > https://bgpstream.com/event/54711 My suggestion is that BackConnect/Bryant Townsend should have their ASN revoked for fraudulently announcing another organization's address space. They are not law enforcement, they did not have a warrant or judicial oversight, they were not in immediate mortal peril, etc, etc. From surfer at mauigateway.com Mon Sep 12 16:08:01 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Mon, 12 Sep 2016 09:08:01 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160912090801.CCA895E5@m0086238.ppops.net> From: NANOG on behalf of Blake Hudson My suggestion is that BackConnect/Bryant Townsend should have their ASN revoked for fraudulently announcing another organization's address space. They are not law enforcement, they did not have a warrant or judicial oversight, they were not in immediate mortal peril, etc, etc. ------------------------------------------------- Are the RIRs the internet police? scott From mel at beckman.org Mon Sep 12 16:09:16 2016 From: mel at beckman.org (Mel Beckman) Date: Mon, 12 Sep 2016 16:09:16 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: , <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net>, Message-ID: <24A3F5DC-112E-4373-A20D-C1E99DC8B27C@beckman.org> Once we let providers cross the line from legal to illegal actions, we're no better than the crooks, and the Internet will descend into lawless chaos. BackConnect's illicit action undoubtedly injured innocent parties, so it's not self defense, any more than shooting wildly into a crowd to stop an attacker would be self defense. This thoughtless action requires a response from the community, and an apology from BackConnect. If we can't police ourselves, someone we don't like will do it for us. -mel beckman > On Sep 12, 2016, at 8:47 AM, Ryan, Spencer wrote: > > I'm in the "never acceptable" camp. Filtering routes/peers? Sure. Disconnecting one of your own customers to stop an attack originating from them? Sure. Hijacking an AS you have no permission to control? No. > > > Obviously my views and not of my employer. > > Spencer Ryan | Senior Systems Administrator | sryan at arbor.net > Arbor Networks > +1.734.794.5033 (d) | +1.734.846.2053 (m) > www.arbornetworks.com > > > ________________________________ > From: NANOG on behalf of Blake Hudson > Sent: Monday, September 12, 2016 11:24:03 AM > To: nanog at nanog.org > Subject: Re: "Defensive" BGP hijacking? > > > Hugo Slabbert wrote on 9/11/2016 3:54 PM: >> Hopefully this is operational enough, though obviously leaning more towards the policy side of things: >> >> What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? >> >> http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ >> >> "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? >> > > > https://bgpstream.com/event/54711 > > My suggestion is that BackConnect/Bryant Townsend should have their ASN > revoked for fraudulently announcing another organization's address > space. They are not law enforcement, they did not have a warrant or > judicial oversight, they were not in immediate mortal peril, etc, etc. From blake at ispn.net Mon Sep 12 16:20:25 2016 From: blake at ispn.net (Blake Hudson) Date: Mon, 12 Sep 2016 11:20:25 -0500 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160912090801.CCA895E5@m0086238.ppops.net> References: <20160912090801.CCA895E5@m0086238.ppops.net> Message-ID: <8d0f8ae4-05c4-48ec-94ca-58b52cf9f373@ispn.net> Scott Weeks wrote on 9/12/2016 11:08 AM: > > > From: NANOG on behalf > of Blake Hudson > > My suggestion is that BackConnect/Bryant Townsend should have their ASN > revoked for fraudulently announcing another organization's address > space. They are not law enforcement, they did not have a warrant or > judicial oversight, they were not in immediate mortal peril, etc, etc. > ------------------------------------------------- > > > Are the RIRs the internet police? > > scott > ARIN has policies against fraudulently obtaining resources and has policies for revoking said resources. One could argue that announcing another org's IP resources without authorization is fraud and that said ip resources were fraudulently obtained during the time they were announced by BlackConnect. That said, this ASN was obtained through RIPE (despite the person/company being located in Calfornia, USA) and I did not see any RIPE policies related to fraud. My thought is that if Mr Townsend shows disregard for the stability of the internet by hijacking other's IP space, he should not be allowed to participate. There are comments to the Kreb's article indicating that this was not an isolated incident by Mr Townsend and instead represents one event in a pattern of behavior. From surfer at mauigateway.com Mon Sep 12 16:31:41 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Mon, 12 Sep 2016 09:31:41 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160912093141.CCA891FD@m0086238.ppops.net> --- blake at ispn.net wrote: From: Blake Hudson Scott Weeks wrote on 9/12/2016 11:08 AM: > From: NANOG on behalf > of Blake Hudson > My suggestion is that BackConnect/Bryant Townsend should have their ASN > revoked for fraudulently announcing another organization's address > space. They are not law enforcement, they did not have a warrant or > judicial oversight, they were not in immediate mortal peril, etc, etc. > ------------------------------------------------- > > > Are the RIRs the internet police? ARIN has policies against fraudulently obtaining resources and has policies for revoking said resources. One could argue that announcing another org's IP resources without authorization is fraud and that said ip resources were fraudulently obtained during the time they were announced by BlackConnect. That said, this ASN was obtained through RIPE (despite the person/company being located in Calfornia, USA) and I did not see any RIPE policies related to fraud. My thought is that if Mr Townsend shows disregard for the stability of the internet by hijacking other's IP space, he should not be allowed to participate. There are comments to the Kreb's article indicating that this was not an isolated incident by Mr Townsend and instead represents one event in a pattern of behavior. ------------------------------------------------- I am somewhat in agreement with Mel: "This thoughtless action requires a response from the community, and an apology from BackConnect. If we can't police ourselves, someone we don't like will do it for us. " But the first part seems to verge on vigilantism. Solutions are hard. BGP filters should be in place. Maybe that's the non-vigilante response. Force filters somehow. However, this has all been discussed over and over here... ;-) scott From jared at puck.nether.net Mon Sep 12 16:42:00 2016 From: jared at puck.nether.net (Jared Mauch) Date: Mon, 12 Sep 2016 12:42:00 -0400 Subject: flag/global cloud exchange 15412 contact Message-ID: <4BB40E61-C61A-4A68-877F-31D95B740D7B@puck.nether.net> Is there someone out here from 15412 I can talk to regarding some BGP related issues? thanks, - jared From hugo at slabnet.com Mon Sep 12 16:51:23 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 12 Sep 2016 09:51:23 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160912093141.CCA891FD@m0086238.ppops.net> References: <20160912093141.CCA891FD@m0086238.ppops.net> Message-ID: <20160912165123.GA16464@bamboo.slabnet.com> On Mon 2016-Sep-12 09:31:41 -0700, Scott Weeks wrote: Full disclosure: I had a working relationship with Bryant when he was still at Staminus. Bryant (if you're on list): I mean no harm by this and never had any trouble working with you. I just believe this is a conversation that needs to be had. >--- blake at ispn.net wrote: >From: Blake Hudson >Scott Weeks wrote on 9/12/2016 11:08 AM: >> From: NANOG on behalf >> of Blake Hudson > > >> My suggestion is that BackConnect/Bryant Townsend should have their ASN >> revoked for fraudulently announcing another organization's address >> space. They are not law enforcement, they did not have a warrant or >> judicial oversight, they were not in immediate mortal peril, etc, etc. >> ------------------------------------------------- >> >> >> Are the RIRs the internet police? > > >ARIN has policies against fraudulently obtaining resources and has >policies for revoking said resources. One could argue that announcing >another org's IP resources without authorization is fraud and that said >ip resources were fraudulently obtained during the time they were >announced by BlackConnect. That said, this ASN was obtained through RIPE >(despite the person/company being located in Calfornia, USA) and I did >not see any RIPE policies related to fraud. > >My thought is that if Mr Townsend shows disregard for the stability of >the internet by hijacking other's IP space, he should not be allowed to >participate. There are comments to the Kreb's article indicating that >this was not an isolated incident by Mr Townsend and instead represents >one event in a pattern of behavior. >------------------------------------------------- > > >I am somewhat in agreement with Mel: > >"This thoughtless action requires a response from the community, and an >apology from BackConnect. If we can't police ourselves, someone we >don't like will do it for us. " > >But the first part seems to verge on vigilantism. Operators are free to do whatever they like inside their own networks as long as they don't impact others. Barring RPKI coverage, we're still talking about an element of trust in BGP to believe what AS 203959 tells us. If I no longer believe what 203959 advertises, I don't have to accept anything with aspath .* 203959 .* in it. I don't see routing policy decisions in my own network as vigilantism. >Solutions are hard. BGP filters should be in place. Maybe that's the >non-vigilante response. Force filters somehow. > >However, this has all been discussed over and over here... ;-) > > >scott -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From blake at ispn.net Mon Sep 12 16:55:14 2016 From: blake at ispn.net (Blake Hudson) Date: Mon, 12 Sep 2016 11:55:14 -0500 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160912093141.CCA891FD@m0086238.ppops.net> References: <20160912093141.CCA891FD@m0086238.ppops.net> Message-ID: <68568a00-8c10-6c64-7f01-335540e93983@ispn.net> Scott Weeks wrote on 9/12/2016 11:31 AM: > > I am somewhat in agreement with Mel: > > "This thoughtless action requires a response from the community, and an > apology from BackConnect. If we can't police ourselves, someone we > don't like will do it for us. " > > But the first part seems to verge on vigilantism. Solutions are hard. > BGP filters should be in place. Maybe that's the non-vigilante response. > Force filters somehow. > > However, this has all been discussed over and over here... ;-) > > > scott I agree that Mel's response is well reasoned and thoughtful. Regarding my mention of a pattern of fraudulent behavior: RIPE indicates that BackConnect has recently announced 55 IP prefixes via BGP (https://stat.ripe.net/widget/as-routing-consistency#w.resource=AS203959), even though they only appear to have 5 IP4 allocations and are currently only announcing 8 /24 prefixes. Given BackConnect's position as an anti-ddos provider it would not be unusual for them to announce the IP space of other organizations. One would likely need to confirm with the owners of each of these 55 prefixes as to whether BackConnect had authorization to announce this address space. Based on the announcement of 82.118.233.0/24, it appears that BGP filters are either not in place for BackConnect or are modified without sufficient procedures to verify authorization. From jcurran at arin.net Mon Sep 12 17:27:09 2016 From: jcurran at arin.net (John Curran) Date: Mon, 12 Sep 2016 17:27:09 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160912090801.CCA895E5@m0086238.ppops.net> References: <20160912090801.CCA895E5@m0086238.ppops.net> Message-ID: <17913CE4-3F88-4ABE-B73F-F5E3F28C6C2D@arin.net> On Sep 12, 2016, at 12:08 PM, Scott Weeks > wrote: Are the RIRs the internet police? Thank you Scott for posing that question? :-) As others have noted, ARIN does indeed revoke resources, but to be clear, this is generally due to fraudulent activities _related_ to the registry itself (i.e. if you commit fraud in the course of obtaining resources, ARIN will revoke those resources once we have determined the fraud beyond reasonable doubt; see ) The specific circumstances raised (of a party announcing an AS# which they do not control) can only happen if the others in the industry allow such, and therefore it is entirely within the community to address. While It is possible that some peering and/or transit agreements have been broken (for example, those agreements which state that the party should only announce routes that they have permission to do so), but in any case, the act of announcing someone else?s number resources stems from usage that the community is allowing to occur, either thru action or inaction, and is not any fraudulent act with respect the Internet number registry itself. ARIN is not a law enforcement entity (although we do work with them on occasion with regard to registry fraud), and it really is up to the industry to ?police? Internet routing to the extent necessary and desirable to keep the Internet running. Thanks, /John John Curran President and CEO ARIN From richard.hesse at weebly.com Mon Sep 12 17:32:18 2016 From: richard.hesse at weebly.com (Richard Hesse) Date: Mon, 12 Sep 2016 17:32:18 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: This behavior is never defensible nor acceptable. In addition to being in the wrong with BGP hijacking a prefix, it appears that Mr. Townsend had the wrong target, too. We've been attacked a few dozen times by this botnet, and they could never muster anything near 200 gbps worth of traffic. They were orders of magnitude smaller, only around 8-16 gbps depending on attack. Mr. Townsend's motives were wrong and so was his information. -richard On Sun, Sep 11, 2016 at 8:54 PM, Hugo Slabbert wrote: > Hopefully this is operational enough, though obviously leaning more towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal From jfmezei_nanog at vaxination.ca Mon Sep 12 17:41:16 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Mon, 12 Sep 2016 13:41:16 -0400 Subject: Lawsuits for falsyfying DNS responses ? Message-ID: <57D6E8BC.9010404@vaxination.ca> As many may know, the province of Qu?bec has passed a law to protect the interests of its lottery corporation. To do so, it will provide ISPs with list of web sites to block (aka: only allow its own gambing web site). There is an opportunity to comment this week in which I will submit. (I've gathered many arguments over the past little while already). But have a specific question today: Are there examples of an ISP getting sued because it redirected traffic that should have gone to original site ? For instance, user asks for www.google.com and ISP's DNS responds with an IP that points to a bing server? If the risk of a lawsuit is real, then it brings new dimension to arguments already made agains that (stupiod) Qu?bec law. (And it also creates interesting issues for DNS servers from companies such as Google which may have a anycast server located in Qu?bec but are not considered an ISP and won't receive those documenst from the gov with list of websites to block. From richard.hesse at weebly.com Mon Sep 12 17:32:18 2016 From: richard.hesse at weebly.com (Richard Hesse) Date: Mon, 12 Sep 2016 17:32:18 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: This behavior is never defensible nor acceptable. In addition to being in the wrong with BGP hijacking a prefix, it appears that Mr. Townsend had the wrong target, too. We've been attacked a few dozen times by this botnet, and they could never muster anything near 200 gbps worth of traffic. They were orders of magnitude smaller, only around 8-16 gbps depending on attack. Mr. Townsend's motives were wrong and so was his information. -richard On Sun, Sep 11, 2016 at 8:54 PM, Hugo Slabbert wrote: > Hopefully this is operational enough, though obviously leaning more towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in-israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting us,? Townsend explained. ?What we were doing was for defensive purposes. We were simply trying to get them to stop and to gather as much information as possible about the botnet they were using and report that to the proper authorities.? > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal From fw at deneb.enyo.de Mon Sep 12 17:59:48 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Mon, 12 Sep 2016 19:59:48 +0200 Subject: "Defensive" BGP hijacking? In-Reply-To: <24A3F5DC-112E-4373-A20D-C1E99DC8B27C@beckman.org> (Mel Beckman's message of "Mon, 12 Sep 2016 16:09:16 +0000") References: <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net> <24A3F5DC-112E-4373-A20D-C1E99DC8B27C@beckman.org> Message-ID: <871t0p3sob.fsf@mid.deneb.enyo.de> * Mel Beckman: > If we can't police ourselves, someone we don't like will do it for us. That hasn't happened with with IP spoofing, has it? As far as I understand it, it is still a major contributing factor in denial-of-service attacks. Self-regulation has been mostly unsuccessful, and yet nothing has happened on the political level. From jfmezei_nanog at vaxination.ca Mon Sep 12 18:07:47 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Mon, 12 Sep 2016 14:07:47 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: <57D6EEF3.90106@vaxination.ca> On 2016-09-11 16:54, Hugo Slabbert wrote: > Hopefully this is operational enough, though obviously leaning more towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? Different spin but still "highjacking": Many moons ago, iStop, a small ISP in Canada saw its services from Bell Canada (access to last mile) cut. However, its core network and transit was still functional for a number of months. ISP2 quickly offered to rescue the stranded customers. Once registred with ISP2, a customer would see the DSL signal re-instated by Bell (now paid by ISP2) but would continue to be handed IPs that belonged to iStop. ISP2 made use of the continuing transit capacity from the iStop router which therefore continued to make BGP announcements for the iStop IP blocks (and the iStop router then just sent everythingt o ISP2's router for distribution to end users). During this time, the iStop IP blocks continued to belong to iStop from ARIn's point of view. Eventually the transit to the iStop router stopped. That day, former iStop customers now on ISP2 saw their access to internet essentially killed. At that point, the iStop IP blocks still had not been transfered to ISP2. To save the day, ISP3 kicked in and started to make BGP annoucements for iStop IPs and redirected the traffic to ISP2. At that point, ISP3 hijacked iStop's IPs, but it was done to help the situation, not to steal traffic or anything. (In fact, I think the GBP announcements from ISP3 pointed to ISP2 routers). Eventually, the iStop IP blocks was transfered to ISP2 which was then legally able to do the BGP announcements for those IPs. So there are some cases where BGP hijacking may be desirable. I guess this is where judgement kicks in. From jared at puck.nether.net Mon Sep 12 18:11:36 2016 From: jared at puck.nether.net (Jared Mauch) Date: Mon, 12 Sep 2016 14:11:36 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <871t0p3sob.fsf@mid.deneb.enyo.de> References: <1a1cb01c-23de-474c-7099-025685fddfd1@ispn.net> <24A3F5DC-112E-4373-A20D-C1E99DC8B27C@beckman.org> <871t0p3sob.fsf@mid.deneb.enyo.de> Message-ID: <52BDC240-4E3D-4102-ADBF-BD534BF57555@puck.nether.net> > On Sep 12, 2016, at 1:59 PM, Florian Weimer wrote: > > * Mel Beckman: > >> If we can't police ourselves, someone we don't like will do it for us. > > That hasn't happened with with IP spoofing, has it? As far as I > understand it, it is still a major contributing factor in > denial-of-service attacks. Self-regulation has been mostly > unsuccessful, and yet nothing has happened on the political level. IP spoofing filtering is more of a technical issue than the social issue of BGP filtering. BGP filtering is feasible in hardware and software today. You can put a 600k line config on most devices without issues, and automate policy generation with a tool like bgpq3 or similar. Most hardware requires a recirculation of the packet to do a lookup on the source IP address. This means halving your NPU performance of something that hasn?t been in the 40 bytes per packet range for quite some time. - Jared From hugo at slabnet.com Mon Sep 12 18:14:02 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 12 Sep 2016 11:14:02 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <57D6EEF3.90106@vaxination.ca> References: <57D6EEF3.90106@vaxination.ca> Message-ID: <20160912181402.GB16464@bamboo.slabnet.com> On Mon 2016-Sep-12 14:07:47 -0400, Jean-Francois Mezei wrote: >On 2016-09-11 16:54, Hugo Slabbert wrote: >> Hopefully this is operational enough, though obviously leaning more towards the policy side of things: >> >> What does nanog think about a DDoS scrubber hijacking a network "for defensive purposes"? > > >Different spin but still "highjacking": > >Many moons ago, iStop, a small ISP in Canada saw its services from Bell >Canada (access to last mile) cut. However, its core network and transit >was still functional for a number of months. > >ISP2 quickly offered to rescue the stranded customers. Once registred >with ISP2, a customer would see the DSL signal re-instated by Bell (now >paid by ISP2) but would continue to be handed IPs that belonged to iStop. > >ISP2 made use of the continuing transit capacity from the iStop router >which therefore continued to make BGP announcements for the iStop IP >blocks (and the iStop router then just sent everythingt o ISP2's router >for distribution to end users). During this time, the iStop IP blocks >continued to belong to iStop from ARIn's point of view. > >Eventually the transit to the iStop router stopped. That day, former >iStop customers now on ISP2 saw their access to internet essentially >killed. At that point, the iStop IP blocks still had not been transfered >to ISP2. > >To save the day, ISP3 kicked in and started to make BGP annoucements for >iStop IPs and redirected the traffic to ISP2. > >At that point, ISP3 hijacked iStop's IPs, but it was done to help the >situation, not to steal traffic or anything. (In fact, I think the GBP >announcements from ISP3 pointed to ISP2 routers). > >Eventually, the iStop IP blocks was transfered to ISP2 which was then >legally able to do the BGP announcements for those IPs. > >So there are some cases where BGP hijacking may be desirable. I guess >this is where judgement kicks in. > Was this all done at iStop's request and with their full support? -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From Valdis.Kletnieks at vt.edu Mon Sep 12 18:15:26 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Mon, 12 Sep 2016 14:15:26 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <57D6EEF3.90106@vaxination.ca> References: <57D6EEF3.90106@vaxination.ca> Message-ID: <5447.1473704126@turing-police.cc.vt.edu> On Mon, 12 Sep 2016 14:07:47 -0400, Jean-Francois Mezei said: > So there are some cases where BGP hijacking may be desirable. I guess > this is where judgement kicks in. I don't see "hijacking" in your description of the iStop case - it appears to have been fully coordinated and with permission. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 830 bytes Desc: not available URL: From mel at beckman.org Mon Sep 12 18:43:48 2016 From: mel at beckman.org (Mel Beckman) Date: Mon, 12 Sep 2016 18:43:48 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: <17913CE4-3F88-4ABE-B73F-F5E3F28C6C2D@arin.net> References: <20160912090801.CCA895E5@m0086238.ppops.net> <17913CE4-3F88-4ABE-B73F-F5E3F28C6C2D@arin.net> Message-ID: <3C92B2DB-A1EE-42C6-B1F0-3735DE3D1E5E@beckman.org> John, I appreciate you making this statement, and I appreciate ARIN?s attitude that this is a community issue. ISPs have done an amazing job of self-regulation, while still preserving their ability to innovate and be agile in the marketplace. BGP is a perfect example of that kind of self-policing. Outside regulation is rarely preferable to community self control, and I think a clear path forward is for those of us in the community to contact BackConnect and respectfully ask that they recognize their incorrect actions and repudiate this practice for the future. Everyone deserves a chance to recognize their mistakes and apologize, so I think we owe BackConnect this much. Nanog seems like a great place for BackConnect to reply to the ISP community as well. -mel > On Sep 12, 2016, at 10:27 AM, John Curran wrote: > > On Sep 12, 2016, at 12:08 PM, Scott Weeks > wrote: > > Are the RIRs the internet police? > > Thank you Scott for posing that question? :-) > > As others have noted, ARIN does indeed revoke resources, but to be clear, > this is generally due to fraudulent activities _related_ to the registry itself > (i.e. if you commit fraud in the course of obtaining resources, ARIN will > revoke those resources once we have determined the fraud beyond > reasonable doubt; see ) > > The specific circumstances raised (of a party announcing an AS# which they > do not control) can only happen if the others in the industry allow such, and > therefore it is entirely within the community to address. While It is possible > that some peering and/or transit agreements have been broken (for example, > those agreements which state that the party should only announce routes that > they have permission to do so), but in any case, the act of announcing someone > else?s number resources stems from usage that the community is allowing to > occur, either thru action or inaction, and is not any fraudulent act with respect > the Internet number registry itself. > > ARIN is not a law enforcement entity (although we do work with them on > occasion with regard to registry fraud), and it really is up to the industry to > ?police? Internet routing to the extent necessary and desirable to keep the > Internet running. > > Thanks, > /John > > John Curran > President and CEO > ARIN > > From jfmezei_nanog at vaxination.ca Mon Sep 12 19:35:13 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Mon, 12 Sep 2016 15:35:13 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160912181402.GB16464@bamboo.slabnet.com> References: <57D6EEF3.90106@vaxination.ca> <20160912181402.GB16464@bamboo.slabnet.com> Message-ID: <57D70371.1000204@vaxination.ca> On 2016-09-12 14:14, Hugo Slabbert wrote: > Was this all done at iStop's request and with their full support? When iStop's router stopped making BGP announcements to the world (because its last transit link was cut), and ISP3 highjacked the IP blocks and made BGP announcements pointing to ISP2, I don't think there was much of iStop left to complain, and it was to the benefit of end users, so this highjacking was not nefarious. Either ISP2 was asleep at the switch and let this happen, or perhaps they had a deal ith iStop that they would not do BGP until block of IPs was transfered, so they got a friend at ISP3 to do the deed for them. The transfer of IP to ISP2 happened shortly after that day, after which ISP2 did the proper BGP announcements for IPs now assigned to it. From jfmezei_nanog at vaxination.ca Mon Sep 12 19:42:02 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Mon, 12 Sep 2016 15:42:02 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <5447.1473704126@turing-police.cc.vt.edu> References: <57D6EEF3.90106@vaxination.ca> <5447.1473704126@turing-police.cc.vt.edu> Message-ID: <57D7050A.6030705@vaxination.ca> On 2016-09-12 14:15, Valdis.Kletnieks at vt.edu wrote: > I don't see "hijacking" in your description of the iStop case - it appears > to have been fully coordinated and with permission. While I am not sure about fully coordinated and with permission, it is an example where it was a desirable outcome to maintain service to customers who would otherwise have have been left without service. I pointed this as an example where "highjacking" can sometimes be desirable. An automated system would likekely block such announcements from ISP3 about ISP1's IP blocks pointing to ISP2's routers as it could be seen as highly suspect. Then again, with many mergers and acquisitions, this type or arrangement may be common as acquiring ISP1 may start to make BGP announcements of ISP2's IPs before those IPs have had time to be transfered. From bill at herrin.us Mon Sep 12 20:08:08 2016 From: bill at herrin.us (William Herrin) Date: Mon, 12 Sep 2016 16:08:08 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D6E8BC.9010404@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> Message-ID: On Mon, Sep 12, 2016 at 1:41 PM, Jean-Francois Mezei wrote: > To do so, it will provide ISPs with list of web sites to block > > Are there examples of an ISP getting sued because it redirected traffic > that should have gone to original site ? Hi, You're talking about two different things here: blocking a DNS domain and redirecting a domain. While both are technologically ineffective countermeasures against undesired content, I would expect the legal implications to be different. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From paras at protrafsolutions.com Mon Sep 12 21:03:06 2016 From: paras at protrafsolutions.com (Paras Jha) Date: Mon, 12 Sep 2016 17:03:06 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: Well don't forget, normal attacks launched from vDOS were around 8 - 16gbps. On the Krebs article, he mentions "the company received an email directly from vDOS claiming credit for the attack" Now, if this holds true, it's likely that the operator of vDOS (Apple J4ck was his moniker) was directing the full resources of the network towards BackConnect. Given that Brian indicated that at any given time vDOS could be launching 10 - 15 times (9 "DDoS years" or something in a few months), the full force of the vDOS network could easily amount to 200gbps. > This behavior is never defensible nor acceptable. > > In addition to being in the wrong with BGP hijacking a prefix, it > appears that Mr. Townsend had the wrong target, too. We've been > attacked a few dozen times by this botnet, and they could never muster > anything near 200 gbps worth of traffic. They were orders of magnitude > smaller, only around 8-16 gbps depending on attack. > > Mr. Townsend's motives were wrong and so was his information. From mel at beckman.org Tue Sep 13 02:08:48 2016 From: mel at beckman.org (Mel Beckman) Date: Tue, 13 Sep 2016 02:08:48 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: , Message-ID: Bryant from BackConnect (bryant at backconnect.com) has replied to me directly. He is a Nanog repeat attendee, but hasn't been subscribed to this list. Bryant says he is subscribing now and will post some clarifying comments shortly. I would share the content of his email, but he didn't explicitly give me permission for that, so I'll let him repeat anything that needs repeating. This looks to me like ISP community governance in the best sense. I look forward to thoughtful discussion. -mel beckman On Sep 12, 2016, at 2:03 PM, Paras Jha > wrote: Well don't forget, normal attacks launched from vDOS were around 8 - 16gbps. On the Krebs article, he mentions "the company received an email directly from vDOS claiming credit for the attack" Now, if this holds true, it's likely that the operator of vDOS (Apple J4ck was his moniker) was directing the full resources of the network towards BackConnect. Given that Brian indicated that at any given time vDOS could be launching 10 - 15 times (9 "DDoS years" or something in a few months), the full force of the vDOS network could easily amount to 200gbps. This behavior is never defensible nor acceptable. In addition to being in the wrong with BGP hijacking a prefix, it appears that Mr. Townsend had the wrong target, too. We've been attacked a few dozen times by this botnet, and they could never muster anything near 200 gbps worth of traffic. They were orders of magnitude smaller, only around 8-16 gbps depending on attack. Mr. Townsend's motives were wrong and so was his information. From surfer at mauigateway.com Tue Sep 13 02:34:01 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Mon, 12 Sep 2016 19:34:01 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160912193401.CCA85158@m0086238.ppops.net> --- mel at beckman.org wrote: From: Mel Beckman This looks to me like ISP community governance in the best sense. I look forward to thoughtful discussion. ------------------------------------------------ Yes, 100% agree! scott From pablo.costa at gmail.com Mon Sep 12 20:00:53 2016 From: pablo.costa at gmail.com (Pablo Costa) Date: Mon, 12 Sep 2016 17:00:53 -0300 Subject: Looking for recommendations for a dedicated ping responder In-Reply-To: <20160909195239.GF3870@dan.olp.net> References: <20160909195239.GF3870@dan.olp.net> Message-ID: Hello Dan, I think that Personar meets your needs Take a look at: http://www.perfsonar.net/about/what-is-perfsonar/ Regards, Pablo On Fri, Sep 9, 2016 at 4:52 PM, Dan White wrote: > Are there any products you're using which are dedicated to responding to > customer facing pings? > > -- > Dan White > BTC Broadband > Network Admin Lead > Ph 918.366.0248 (direct) main: (918)366-8000 > Fax 918.366.6610 email: dwhite at olp.net > http://www.btcbroadband.com > From marcus at blazingdot.com Tue Sep 13 01:38:47 2016 From: marcus at blazingdot.com (Marcus Reid) Date: Mon, 12 Sep 2016 18:38:47 -0700 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D6E8BC.9010404@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> Message-ID: <20160913013847.GA582@blazingdot.com> On Mon, Sep 12, 2016 at 01:41:16PM -0400, Jean-Francois Mezei wrote: > Are there examples of an ISP getting sued because it redirected traffic > that should have gone to original site ? This happened with Paxfire (and the ISPs that used them) in 2011. https://www.eff.org/deeplinks/2011/07/widespread-search-hijacking-in-the-us http://www.courthousenews.com/2011/08/08/38796.htm Marcus From jako.andras at eik.bme.hu Tue Sep 13 05:12:59 2016 From: jako.andras at eik.bme.hu (=?utf-8?B?SsOBS8OTIEFuZHLDoXM=?=) Date: Tue, 13 Sep 2016 07:12:59 +0200 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: References: <57D6E8BC.9010404@vaxination.ca> Message-ID: <20160913051258.GA563@eik.bme.hu> On Mon, Sep 12, 2016 at 04:08:08PM -0400, William Herrin wrote: > On Mon, Sep 12, 2016 at 1:41 PM, Jean-Francois Mezei > wrote: > > To do so, it will provide ISPs with list of web sites to block > > > > Are there examples of an ISP getting sued because it redirected traffic > > that should have gone to original site ? > > Hi, > > You're talking about two different things here: blocking a DNS domain > and redirecting a domain. Blocking for that purpose usually means redirecting in practive. You'll redirect to a page that explains why the original site is not available. Andr?s From janusz at speedchecker.xyz Tue Sep 13 07:42:13 2016 From: janusz at speedchecker.xyz (Janusz Jezowicz) Date: Tue, 13 Sep 2016 09:42:13 +0200 Subject: Akamai Edgescape query Message-ID: Hello, We are doing some network research to various cloud/CDNs and we need to run few tests on Akamai Edgescape. Does anyone has access to run few queries to geo-locate IPs using Edgescape for us? We don't have Akamai contract and it would take us lot of time (and money) to get one just to run few tests. Many thanks for any help! Janusz Jezowicz *Speedchecker Ltd* *email*: janusz at speedchecker.xyz *skype*: jezowicz *phone*: +442032863573 *web*: www.speedchecker.xyz The Black Church, St. Mary?s Place, Dublin 7, D07 P4AX, Ireland From bortzmeyer at nic.fr Tue Sep 13 08:30:24 2016 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Tue, 13 Sep 2016 10:30:24 +0200 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <20160913051258.GA563@eik.bme.hu> References: <57D6E8BC.9010404@vaxination.ca> <20160913051258.GA563@eik.bme.hu> Message-ID: <20160913083024.aig5upvt4zlvqhmv@nic.fr> On Tue, Sep 13, 2016 at 07:12:59AM +0200, J?K? Andr?s wrote a message of 18 lines which said: > Blocking for that purpose usually means redirecting in > practive. You'll redirect to a page that explains why the original > site is not available. It has practical consequences for the user: in France, DNS lies in ISP's resolvers for "terrorist" sites redirect you to a Web site of the police, which will get your source IP address and the site you wanted (thanks to the Host: HTTP field). Clear blocking (DNS lie returning localhost or NXDOMAIN) is a bit better for privacy. (But less transparent about the censorship.) From ahebert at pubnix.net Tue Sep 13 12:29:25 2016 From: ahebert at pubnix.net (Alain Hebert) Date: Tue, 13 Sep 2016 08:29:25 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D6E8BC.9010404@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> Message-ID: Well "may" is not "must". ?260.34. An Internet service provider may not give access to an online gambling site whose operation is not authorized under Qu?bec law. ----- Alain Hebert ahebert at pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 09/12/16 13:41, Jean-Francois Mezei wrote: > As many may know, the province of Qu?bec has passed a law to protect the > interests of its lottery corporation. > > To do so, it will provide ISPs with list of web sites to block (aka: > only allow its own gambing web site). > > There is an opportunity to comment this week in which I will submit. > > (I've gathered many arguments over the past little while already). But > have a specific question today: > > Are there examples of an ISP getting sued because it redirected traffic > that should have gone to original site ? > > For instance, user asks for www.google.com and ISP's DNS responds with > an IP that points to a bing server? > > If the risk of a lawsuit is real, then it brings new dimension to > arguments already made agains that (stupiod) Qu?bec law. > > (And it also creates interesting issues for DNS servers from companies > such as Google which may have a anycast server located in Qu?bec but are > not considered an ISP and won't receive those documenst from the gov > with list of websites to block. > > From deleskie at gmail.com Tue Sep 13 12:31:12 2016 From: deleskie at gmail.com (jim deleskie) Date: Tue, 13 Sep 2016 09:31:12 -0300 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: Redirecting someone's traffic, with out there permission or a court order, by a court in your jurisdiction, not a lot different then the "bad guys" themselves. On Sun, Sep 11, 2016 at 5:54 PM, Hugo Slabbert wrote: > Hopefully this is operational enough, though obviously leaning more > towards the policy side of things: > > What does nanog think about a DDoS scrubber hijacking a network "for > defensive purposes"? > > http://krebsonsecurity.com/2016/09/alleged-vdos-proprietors-arrested-in- > israel/ > > "For about six hours, we were seeing attacks of more than 200 Gbps hitting > us,? Townsend explained. ?What we were doing was for defensive purposes. We > were simply trying to get them to stop and to gather as much information as > possible about the botnet they were using and report that to the proper > authorities.? > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal From Valdis.Kletnieks at vt.edu Tue Sep 13 14:30:53 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Tue, 13 Sep 2016 10:30:53 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: References: <57D6E8BC.9010404@vaxination.ca> Message-ID: <147637.1473777053@turing-police.cc.vt.edu> On Tue, 13 Sep 2016 08:29:25 -0400, Alain Hebert said: > Well "may" is not "must". > > ???260.34. An Internet service provider may not give access to an online > gambling site whose operation is not authorized under Qu?bec law. Note that most legal jurisdictions don't include RFC2119 as part of their legal language. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 830 bytes Desc: not available URL: From zvernhout at gmail.com Tue Sep 13 14:44:15 2016 From: zvernhout at gmail.com (Matthew Vernhout) Date: Tue, 13 Sep 2016 10:44:15 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D6E8BC.9010404@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> Message-ID: Jean-Francois, Canada's Anti-Spam Legislation has specific sections that makes altering of data illegal under the Act. In my non-lawyer opinion, sections 10 (5) (b) and (e) would be violated by hijacking someone preference to go to Website A and replace it with Website B without their express consent to do so. Source: http://laws-lois.justice.gc.ca/PDF/E-1.6.pdf *Section 10 - 5 * Description of functions (5) A function referred to in subsection (4) is any of the following functions that the person who seeks express consent knows and intends will cause the computer system to operate in a manner that is contrary to the reasonable expectations of the owner or an authorized user of the computer system: (a) collecting personal information stored on the computer system; *(b) interfering with the owner?s or an authorized user?s control of the computer system; * (c) changing or interfering with settings, preferences or commands already installed or stored on the computer system without the knowledge of the owner or an authorized user of the computer system; (d) changing or interfering with data that is stored on the computer system in a manner that obstructs, interrupts or interferes with lawful access to or use of that data by the owner or an authorized user of the computer system; *(e) causing the computer system to communicate with another computer system, or other device, without the authorization of the owner or an authorized user of the computer system; * (f) installing a computer program that may be activated by a third party without the knowledge of the owner or an authorized user of the computer system; and (g) performing any other function specified in the regulations. It might be interesting to bring this to their attention, or the attention of your own lawyers for comment. Cheers, ~ Matt On 12/09/2016 1:41 PM, Jean-Francois Mezei wrote: > As many may know, the province of Qu?bec has passed a law to protect the > interests of its lottery corporation. > > To do so, it will provide ISPs with list of web sites to block (aka: > only allow its own gambing web site). > > There is an opportunity to comment this week in which I will submit. > > (I've gathered many arguments over the past little while already). But > have a specific question today: > > Are there examples of an ISP getting sued because it redirected traffic > that should have gone to original site ? > > For instance, user asks for www.google.com and ISP's DNS responds with > an IP that points to a bing server? > > If the risk of a lawsuit is real, then it brings new dimension to > arguments already made agains that (stupiod) Qu?bec law. > > (And it also creates interesting issues for DNS servers from companies > such as Google which may have a anycast server located in Qu?bec but are > not considered an ISP and won't receive those documenst from the gov > with list of websites to block. > From johnl at iecc.com Tue Sep 13 14:58:43 2016 From: johnl at iecc.com (John Levine) Date: 13 Sep 2016 14:58:43 -0000 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: Message-ID: <20160913145843.53307.qmail@ary.lan> In article you write: >Canada's Anti-Spam Legislation has specific sections that makes altering >of data illegal under the Act. > >In my non-lawyer opinion, sections 10 (5) (b) and (e) would be violated >by hijacking someone preference to go to Website A and replace it with >Website B without their express consent to do so. That section only applies to 10(4) which is about getting permission to install downloaded software. >Description of functions > >(5) A function referred to in subsection (4) is any of the following >functions ... I don't think the Quebec law is a good idea, or is likely to be effective, but I also don't think it has preemption issues. R's, John From bryant at backconnect.com Tue Sep 13 07:22:43 2016 From: bryant at backconnect.com (Bryant Townsend) Date: Tue, 13 Sep 2016 00:22:43 -0700 Subject: "Defensive" BGP hijacking? Message-ID: Hello Everyone, I would like to give as much insight as I can in regards to the BGP hijack being discussed in this thread. I won?t be going into specific details of the attack, but we do plan to release more information on our website when we are able to. I also wanted to let Hugo (who started the thread) know that we harbor no hard feelings about bringing this topic up, as it is relevant to the community and does warrant discussion. Hugo, you may owe me a beer the next time we meet. :) We agree with others that NANOG is the most appropriate venue to answer any questions and discuss the topic at hand. I have been attending NANOG for the past 3-4 years, and I can assure you that it is of the utmost importance to me how the community views my company, my employees, and myself. There are many people in this community that I personally have the upmost respect for, and it would sadden me If I were to lose the respect of mentors, colleagues, and friends by not responding. That being said, I think there are a fair number of people in NANOG that would vouch for my character and ethics relating to the intent of my actions, even if I were to remain silent. I would also like to preface that my explanation of the events that occurred and actions taken by BackConnect are not to justify or provide excuses. My goal is to simply show what happened and give insight into our actions. I will start with a little background to bring anyone up to speed that is not aware of the events that transpired. *About the company, BackConnect, Inc.*: We are a new (~4 months old) open-sourced based DDoS mitigation and network security provider that specializes in custom intrusion detection and prevention systems. We also provide threat intelligence services, with an emphasis on active botnets, new and upcoming DDoS attack patterns, and boot services. From time to time, this information flows through our network for collection purposes. *Events leading to the Hijack*: On 9/6/2016, ~10:30AM PST, one of our clients and our website received a large and relatively sophisticated DDoS attack. The attack targeted entire subnets and peaked over 200 Gbps and 40Mpps. Although the attack was automatically detected and mostly filtered, there was initially a small leak. In response we quickly applied new security rules that rendered it entirely ineffective. The attackers continued to attack our network and client for roughly 6 hours before giving up. *Events that caused us to perform the BGP hijack*: After the DDoS attacks subsided, the attackers started to harass us by calling in using spoofed phone numbers. Curious to what this was all about, we fielded various calls which allowed us to ascertain who was behind the attacks by correlating e-mails with the information they provided over the phone. Throughout the day and late into the night, these calls and threats continued to increase in number. Throughout these calls we noticed an increasing trend of them bringing up personal information of myself and employees. At this point I personally filled a police report in preparation to a possible SWATing attempt. As they continued to harass our company, more and more red flags indicated that I would soon be targeted. This was the point where I decided I needed to go on the offensive to protect myself, my partner, visiting family, and my employees. The actions proved to be extremely effective, as all forms of harassment and threats from the attackers immediately stopped. In addition to our main objective, we were able to collect intelligence on the actors behind the bot net as well as identify the attack servers used by the booter service. *Afterthoughts*: The decision to hijack the attackers IP space was not something I took lightly. I was fully aware there were services that reported such actions and knew that this could potentially be brought up in discussion and hurt BackConnect?s image. Even though we had the capacity to hide our actions, we felt that it would be wrong to do so. I have spent a long time reflecting on my decision and how it may negatively impact the company and myself in some people?s eyes, but ultimately I stand by it. The experience and feedback I have gained from these events has proven invaluable and will be used to shape the policies surrounding the future handling of similar situations. I am happy to field questions, but cannot promise any answers, disclosure of further information, or when they will be responded to. Sincerely, Bryant Townsend From large.hadron.collider at gmx.com Tue Sep 13 07:42:04 2016 From: large.hadron.collider at gmx.com (LHC) Date: Tue, 13 Sep 2016 07:42:04 +0000 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D6E8BC.9010404@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> Message-ID: I believe that the CRTC has rules against censorship - meaning that Videotron, Bell etcetera have a choice between following the CRTC code or the provincial law (following one = sanctions from the other), rendering internet service provision to Qu?bec impossible without being a dialup provider from out-of-province. The law may even be actually contrary to federal law. On September 12, 2016 10:41:16 AM PDT, Jean-Francois Mezei wrote: >As many may know, the province of Qu?bec has passed a law to protect >the >interests of its lottery corporation. > >To do so, it will provide ISPs with list of web sites to block (aka: >only allow its own gambing web site). > >There is an opportunity to comment this week in which I will submit. > >(I've gathered many arguments over the past little while already). But >have a specific question today: > >Are there examples of an ISP getting sued because it redirected traffic >that should have gone to original site ? > >For instance, user asks for www.google.com and ISP's DNS responds with >an IP that points to a bing server? > >If the risk of a lawsuit is real, then it brings new dimension to >arguments already made agains that (stupiod) Qu?bec law. > >(And it also creates interesting issues for DNS servers from companies >such as Google which may have a anycast server located in Qu?bec but >are >not considered an ISP and won't receive those documenst from the gov >with list of websites to block. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From cb.list6 at gmail.com Tue Sep 13 16:22:29 2016 From: cb.list6 at gmail.com (Ca By) Date: Tue, 13 Sep 2016 09:22:29 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: On Tuesday, September 13, 2016, Bryant Townsend wrote: > Hello Everyone, > > > I would like to give as much insight as I can in regards to the BGP hijack > being discussed in this thread. I won?t be going into specific details of > the attack, but we do plan to release more information on our website when > we are able to. I also wanted to let Hugo (who started the thread) know > that we harbor no hard feelings about bringing this topic up, as it is > relevant to the community and does warrant discussion. Hugo, you may owe me > a beer the next time we meet. :) > > > > We agree with others that NANOG is the most appropriate venue to answer any > questions and discuss the topic at hand. I have been attending NANOG for > the past 3-4 years, and I can assure you that it is of the utmost > importance to me how the community views my company, my employees, and > myself. There are many people in this community that I personally have the > upmost respect for, and it would sadden me If I were to lose the respect of > mentors, colleagues, and friends by not responding. That being said, I > think there are a fair number of people in NANOG that would vouch for my > character and ethics relating to the intent of my actions, even if I were > to remain silent. I would also like to preface that my explanation of the > events that occurred and actions taken by BackConnect are not to justify or > provide excuses. My goal is to simply show what happened and give insight > into our actions. > > > > I will start with a little background to bring anyone up to speed that is > not aware of the events that transpired. > > > *About the company, BackConnect, Inc.*: We are a new (~4 months old) > open-sourced based DDoS mitigation and network security provider that > specializes in custom intrusion detection and prevention systems. We also > provide threat intelligence services, with an emphasis on active botnets, > new and upcoming DDoS attack patterns, and boot services. From time to > time, this information flows through our network for collection purposes. > > > *Events leading to the Hijack*: On 9/6/2016, ~10:30AM PST, one of our > clients and our website received a large and relatively sophisticated DDoS > attack. The attack targeted entire subnets and peaked over 200 Gbps and > 40Mpps. Although the attack was automatically detected and mostly filtered, > there was initially a small leak. In response we quickly applied new > security rules that rendered it entirely ineffective. The attackers > continued to attack our network and client for roughly 6 hours before > giving up. > > > *Events that caused us to perform the BGP hijack*: After the DDoS attacks > subsided, the attackers started to harass us by calling in using spoofed > phone numbers. Curious to what this was all about, we fielded various calls > which allowed us to ascertain who was behind the attacks by correlating > e-mails with the information they provided over the phone. Throughout the > day and late into the night, these calls and threats continued to increase > in number. Throughout these calls we noticed an increasing trend of them > bringing up personal information of myself and employees. At this point I > personally filled a police report in preparation to a possible SWATing > attempt. As they continued to harass our company, more and more red flags > indicated that I would soon be targeted. This was the point where I decided > I needed to go on the offensive to protect myself, my partner, visiting > family, and my employees. The actions proved to be extremely effective, as > all forms of harassment and threats from the attackers immediately stopped. > In addition to our main objective, we were able to collect intelligence on > the actors behind the bot net as well as identify the attack servers used > by the booter service. > > > > *Afterthoughts*: The decision to hijack the attackers IP space was not > something I took lightly. I was fully aware there were services that > reported such actions and knew that this could potentially be brought up in > discussion and hurt BackConnect?s image. Even though we had the capacity to > hide our actions, we felt that it would be wrong to do so. I have spent a > long time reflecting on my decision and how it may negatively impact the > company and myself in some people?s eyes, but ultimately I stand by it. The > experience and feedback I have gained from these events has proven > invaluable and will be used to shape the policies surrounding the future > handling of similar situations. I am happy to field questions, but cannot > promise any answers, disclosure of further information, or when they will > be responded to. > > > Sincerely, > > Bryant Townsend > Will you do the bgp hijacking in the future: yes or no? Thanks! From mlfreita at mtu.edu Tue Sep 13 18:25:36 2016 From: mlfreita at mtu.edu (Matt Freitag) Date: Tue, 13 Sep 2016 14:25:36 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: +1 to this question. Bryant, thanks for giving us your side of this story. Matt Freitag Network Engineer I Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.it.mtu.edu/ On Tue, Sep 13, 2016 at 12:22 PM, Ca By wrote: > On Tuesday, September 13, 2016, Bryant Townsend > wrote: > > > Hello Everyone, > > > > > > I would like to give as much insight as I can in regards to the BGP > hijack > > being discussed in this thread. I won?t be going into specific details of > > the attack, but we do plan to release more information on our website > when > > we are able to. I also wanted to let Hugo (who started the thread) know > > that we harbor no hard feelings about bringing this topic up, as it is > > relevant to the community and does warrant discussion. Hugo, you may owe > me > > a beer the next time we meet. :) > > > > > > > > We agree with others that NANOG is the most appropriate venue to answer > any > > questions and discuss the topic at hand. I have been attending NANOG for > > the past 3-4 years, and I can assure you that it is of the utmost > > importance to me how the community views my company, my employees, and > > myself. There are many people in this community that I personally have > the > > upmost respect for, and it would sadden me If I were to lose the respect > of > > mentors, colleagues, and friends by not responding. That being said, I > > think there are a fair number of people in NANOG that would vouch for my > > character and ethics relating to the intent of my actions, even if I were > > to remain silent. I would also like to preface that my explanation of > the > > events that occurred and actions taken by BackConnect are not to justify > or > > provide excuses. My goal is to simply show what happened and give insight > > into our actions. > > > > > > > > I will start with a little background to bring anyone up to speed that is > > not aware of the events that transpired. > > > > > > *About the company, BackConnect, Inc.*: We are a new (~4 months old) > > open-sourced based DDoS mitigation and network security provider that > > specializes in custom intrusion detection and prevention systems. We also > > provide threat intelligence services, with an emphasis on active botnets, > > new and upcoming DDoS attack patterns, and boot services. From time to > > time, this information flows through our network for collection purposes. > > > > > > *Events leading to the Hijack*: On 9/6/2016, ~10:30AM PST, one of our > > clients and our website received a large and relatively sophisticated > DDoS > > attack. The attack targeted entire subnets and peaked over 200 Gbps and > > 40Mpps. Although the attack was automatically detected and mostly > filtered, > > there was initially a small leak. In response we quickly applied new > > security rules that rendered it entirely ineffective. The attackers > > continued to attack our network and client for roughly 6 hours before > > giving up. > > > > > > *Events that caused us to perform the BGP hijack*: After the DDoS attacks > > subsided, the attackers started to harass us by calling in using spoofed > > phone numbers. Curious to what this was all about, we fielded various > calls > > which allowed us to ascertain who was behind the attacks by correlating > > e-mails with the information they provided over the phone. Throughout the > > day and late into the night, these calls and threats continued to > increase > > in number. Throughout these calls we noticed an increasing trend of them > > bringing up personal information of myself and employees. At this point I > > personally filled a police report in preparation to a possible SWATing > > attempt. As they continued to harass our company, more and more red > flags > > indicated that I would soon be targeted. This was the point where I > decided > > I needed to go on the offensive to protect myself, my partner, visiting > > family, and my employees. The actions proved to be extremely effective, > as > > all forms of harassment and threats from the attackers immediately > stopped. > > In addition to our main objective, we were able to collect intelligence > on > > the actors behind the bot net as well as identify the attack servers used > > by the booter service. > > > > > > > > *Afterthoughts*: The decision to hijack the attackers IP space was not > > something I took lightly. I was fully aware there were services that > > reported such actions and knew that this could potentially be brought up > in > > discussion and hurt BackConnect?s image. Even though we had the capacity > to > > hide our actions, we felt that it would be wrong to do so. I have spent a > > long time reflecting on my decision and how it may negatively impact the > > company and myself in some people?s eyes, but ultimately I stand by it. > The > > experience and feedback I have gained from these events has proven > > invaluable and will be used to shape the policies surrounding the future > > handling of similar situations. I am happy to field questions, but cannot > > promise any answers, disclosure of further information, or when they will > > be responded to. > > > > > > Sincerely, > > > > Bryant Townsend > > > > > Will you do the bgp hijacking in the future: yes or no? > > Thanks! > From sryan at arbor.net Tue Sep 13 18:28:19 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Tue, 13 Sep 2016 18:28:19 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: What would you have done if the personal harassment didn't stop? What would you have done if they simply switched to a new source range/different set of bots? Seems like a very slippery slope to me. Spencer Ryan | Senior Systems Administrator | sryan at arbor.net Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com ________________________________ From: NANOG on behalf of Bryant Townsend Sent: Tuesday, September 13, 2016 3:22:43 AM To: nanog at nanog.org Subject: Re: "Defensive" BGP hijacking? Hello Everyone, I would like to give as much insight as I can in regards to the BGP hijack being discussed in this thread. I won?t be going into specific details of the attack, but we do plan to release more information on our website when we are able to. I also wanted to let Hugo (who started the thread) know that we harbor no hard feelings about bringing this topic up, as it is relevant to the community and does warrant discussion. Hugo, you may owe me a beer the next time we meet. :) We agree with others that NANOG is the most appropriate venue to answer any questions and discuss the topic at hand. I have been attending NANOG for the past 3-4 years, and I can assure you that it is of the utmost importance to me how the community views my company, my employees, and myself. There are many people in this community that I personally have the upmost respect for, and it would sadden me If I were to lose the respect of mentors, colleagues, and friends by not responding. That being said, I think there are a fair number of people in NANOG that would vouch for my character and ethics relating to the intent of my actions, even if I were to remain silent. I would also like to preface that my explanation of the events that occurred and actions taken by BackConnect are not to justify or provide excuses. My goal is to simply show what happened and give insight into our actions. I will start with a little background to bring anyone up to speed that is not aware of the events that transpired. *About the company, BackConnect, Inc.*: We are a new (~4 months old) open-sourced based DDoS mitigation and network security provider that specializes in custom intrusion detection and prevention systems. We also provide threat intelligence services, with an emphasis on active botnets, new and upcoming DDoS attack patterns, and boot services. From time to time, this information flows through our network for collection purposes. *Events leading to the Hijack*: On 9/6/2016, ~10:30AM PST, one of our clients and our website received a large and relatively sophisticated DDoS attack. The attack targeted entire subnets and peaked over 200 Gbps and 40Mpps. Although the attack was automatically detected and mostly filtered, there was initially a small leak. In response we quickly applied new security rules that rendered it entirely ineffective. The attackers continued to attack our network and client for roughly 6 hours before giving up. *Events that caused us to perform the BGP hijack*: After the DDoS attacks subsided, the attackers started to harass us by calling in using spoofed phone numbers. Curious to what this was all about, we fielded various calls which allowed us to ascertain who was behind the attacks by correlating e-mails with the information they provided over the phone. Throughout the day and late into the night, these calls and threats continued to increase in number. Throughout these calls we noticed an increasing trend of them bringing up personal information of myself and employees. At this point I personally filled a police report in preparation to a possible SWATing attempt. As they continued to harass our company, more and more red flags indicated that I would soon be targeted. This was the point where I decided I needed to go on the offensive to protect myself, my partner, visiting family, and my employees. The actions proved to be extremely effective, as all forms of harassment and threats from the attackers immediately stopped. In addition to our main objective, we were able to collect intelligence on the actors behind the bot net as well as identify the attack servers used by the booter service. *Afterthoughts*: The decision to hijack the attackers IP space was not something I took lightly. I was fully aware there were services that reported such actions and knew that this could potentially be brought up in discussion and hurt BackConnect?s image. Even though we had the capacity to hide our actions, we felt that it would be wrong to do so. I have spent a long time reflecting on my decision and how it may negatively impact the company and myself in some people?s eyes, but ultimately I stand by it. The experience and feedback I have gained from these events has proven invaluable and will be used to shape the policies surrounding the future handling of similar situations. I am happy to field questions, but cannot promise any answers, disclosure of further information, or when they will be responded to. Sincerely, Bryant Townsend From blake at ispn.net Tue Sep 13 18:39:31 2016 From: blake at ispn.net (Blake Hudson) Date: Tue, 13 Sep 2016 13:39:31 -0500 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: Bryant Townsend wrote on 9/13/2016 2:22 AM: > This was the point where I decided > I needed to go on the offensive to protect myself, my partner, visiting > family, and my employees. The actions proved to be extremely effective, as > all forms of harassment and threats from the attackers immediately stopped. Bryant, what actions, exactly, did you take? This topic seems intentionally glossed over while you spend a much larger amount of time explaining the back story and your motivations rather than your actions. Questions I was left with: 1. What prefixes have you announced without permission (not just this event)? 2. How did you identify these prefixes? 3. Did you attempt to contact the owner of these prefixes? 4. Did you attempt to contact the origin or transit AS of these prefixes? 5. What was the process to get your upstream AS to accept these prefix announcements? 6. Was your upstream AS complicit in allowing you to announce prefixes you did not have authorization to announce? From steve at blighty.com Tue Sep 13 18:47:56 2016 From: steve at blighty.com (Steve Atkins) Date: Tue, 13 Sep 2016 11:47:56 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> > On Sep 13, 2016, at 12:22 AM, Bryant Townsend wrote: > > *Events that caused us to perform the BGP hijack*: After the DDoS attacks > subsided, the attackers started to harass us by calling in using spoofed > phone numbers. Curious to what this was all about, we fielded various calls > which allowed us to ascertain who was behind the attacks by correlating > e-mails with the information they provided over the phone. Throughout the > day and late into the night, these calls and threats continued to increase > in number. Throughout these calls we noticed an increasing trend of them > bringing up personal information of myself and employees. At this point I > personally filled a police report in preparation to a possible SWATing > attempt. As they continued to harass our company, more and more red flags > indicated that I would soon be targeted. This was the point where I decided > I needed to go on the offensive to protect myself, my partner, visiting > family, and my employees. I think you're saying that the BGP hijack wasn't done in as part of an attempt to mitigate a DDoS, rather that you used the tools you had available to go on the offensive in response to phone calls you received. Am I reading that right? Cheers, Steve From mel at beckman.org Tue Sep 13 18:51:06 2016 From: mel at beckman.org (Mel Beckman) Date: Tue, 13 Sep 2016 18:51:06 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: , Message-ID: <68875853-EB91-4BAB-9116-BAF7B28E5117@beckman.org> Blake, I concur that these are key questions. Probably _the_ key questions. The fabric of the Internet is today based on trust, and BGP's integrity is the core of that trust. I realize that BGP hijacking is not uncommon. However, this is the first time I've seen in it used defensively. I don't see a way to ever bless this kind of defensive use without compromising that core trust. If Internet reachability depends on individual providers believing that they are justified in violating that trust when they are attacked, how can the Internet stand? In addition to the question posed to Bryant about whether he would take this action again, I would like to add: what about the innocent parties impacted by your actions? Or do you take the position there were no innocent parties in the hijacked prefixes? -mel via cell > On Sep 13, 2016, at 11:40 AM, Blake Hudson wrote: > > > > Bryant Townsend wrote on 9/13/2016 2:22 AM: >> This was the point where I decided >> I needed to go on the offensive to protect myself, my partner, visiting >> family, and my employees. The actions proved to be extremely effective, as >> all forms of harassment and threats from the attackers immediately stopped. > > > Bryant, what actions, exactly, did you take? This topic seems intentionally glossed over while you spend a much larger amount of time explaining the back story and your motivations rather than your actions. > > Questions I was left with: > > 1. What prefixes have you announced without permission (not just this > event)? > 2. How did you identify these prefixes? > 3. Did you attempt to contact the owner of these prefixes? > 4. Did you attempt to contact the origin or transit AS of these prefixes? > 5. What was the process to get your upstream AS to accept these prefix > announcements? > 6. Was your upstream AS complicit in allowing you to announce prefixes > you did not have authorization to announce? > From hf0002+nanog at uah.edu Tue Sep 13 18:56:45 2016 From: hf0002+nanog at uah.edu (Hunter Fuller) Date: Tue, 13 Sep 2016 18:56:45 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: On Tue, Sep 13, 2016 at 10:25 AM Bryant Townsend wrote: > I also wanted to let Hugo (who started the thread) know > that we harbor no hard feelings about bringing this topic up, as it is > relevant to the community and does warrant discussion. Hugo, you may owe me > a beer the next time we meet. :) > Wait, so if I hijack someone else's prefix, someone ends up buying me a beer? Let me fire up my terminal... From bryant at backconnect.com Tue Sep 13 19:29:30 2016 From: bryant at backconnect.com (Bryant Townsend) Date: Tue, 13 Sep 2016 12:29:30 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> Message-ID: @ca & Matt - No, we do not plan to ever intentionally perform a non-authorized BGP hijack in the future. @Steve - Correct, the attack had already been mitigated. The decision to hijack the attackers IP space was to deal with their threats, which if carried through could have potentially lead to physical harm. Although the hijack gave us a unique insight into the attackers services, it was not a factor that influenced my decision. @Blake & Mel - We will likely cover some of these questions in a future blog post. From cb.list6 at gmail.com Tue Sep 13 19:53:17 2016 From: cb.list6 at gmail.com (Ca By) Date: Tue, 13 Sep 2016 12:53:17 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> Message-ID: On Tuesday, September 13, 2016, Bryant Townsend wrote: > @ca & Matt - No, we do not plan to ever intentionally perform a > non-authorized BGP hijack in the future. > > Great answer. Thanks. Committing to pursuing a policy of weaponizing BGP would have triggered a serious "terms of service" violations that would have effectively ended your business swiftly and permanently. Tip to the RIR policy folks, you may want to make this point very crisp. A BGP ASN is the fundamental accountability control in a inter-domain routing. Organizations with repeated offensense need to have their ASN revoked, and further there should be controls in places so bad actors cannot acquire "burner" ASNs. @Steve - Correct, the attack had already been mitigated. The decision to > hijack the attackers IP space was to deal with their threats, which if > carried through could have potentially lead to physical harm. Although the > hijack gave us a unique insight into the attackers services, it was not a > factor that influenced my decision. > > @Blake & Mel - We will likely cover some of these questions in a future > blog post. > From blake at ispn.net Tue Sep 13 20:22:10 2016 From: blake at ispn.net (Blake Hudson) Date: Tue, 13 Sep 2016 15:22:10 -0500 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> Message-ID: <20f702e2-913a-5c03-8ba2-4513faceb228@ispn.net> Ca By wrote on 9/13/2016 2:53 PM: > On Tuesday, September 13, 2016, Bryant Townsend > wrote: > >> @ca & Matt - No, we do not plan to ever intentionally perform a >> non-authorized BGP hijack in the future. > Great answer. Thanks. > > Committing to pursuing a policy of weaponizing BGP would have triggered a > serious "terms of service" violations that would have effectively ended > your business swiftly and permanently. > > Tip to the RIR policy folks, you may want to make this point very crisp. A > BGP ASN is the fundamental accountability control in a inter-domain > routing. Organizations with repeated offensense need to have their ASN > revoked, and further there should be controls in places so bad actors > cannot acquire "burner" ASNs. > >> @Steve - Correct, the attack had already been mitigated. The decision to >> hijack the attackers IP space was to deal with their threats, which if >> carried through could have potentially lead to physical harm. Although the >> hijack gave us a unique insight into the attackers services, it was not a >> factor that influenced my decision. >> >> @Blake & Mel - We will likely cover some of these questions in a future >> blog post. >> Ca, and the community, I don't make the leap. How does attacking someone by hijacking their IP space mitigate a physical threat? How does impeding someone's access to the internet access prevent them from performing an act of physical violence against you? If a party threatens me, would I be justified in attacking him or her? In my experience, attacking someone is more likely to escalate a situation - not deescalate it. Bryant did weaponize BGP and stated he stands by his actions and further indicated that he will use what he learned here to shape handling of future situations: > I have spent a > long time reflecting on my decision and how it may negatively impact the > company and myself in some people?s eyes, but ultimately I stand by it. The > experience and feedback I have gained from these events has proven > invaluable and will be used to shape the policies surrounding the future > handling of similar situations. When I read Bryant's comments, I see justification and excuses for his behavior. I do not see an apology nor admission of wrongdoing. I believe what Bryant did was wrong and I would hate for others to be allowed to act similarly without consequence. From surfer at mauigateway.com Tue Sep 13 20:32:56 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Tue, 13 Sep 2016 13:32:56 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160913133256.7605C006@m0087791.ppops.net> --- bryant at backconnect.com wrote: From: Bryant Townsend @ca & Matt - No, we do not plan to ever intentionally perform a non-authorized BGP hijack in the future. -------------------------------------------- Bryant, Who was the upstream provider? scott From hugo at slabnet.com Tue Sep 13 20:36:19 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Tue, 13 Sep 2016 13:36:19 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160913133256.7605C006@m0087791.ppops.net> References: <20160913133256.7605C006@m0087791.ppops.net> Message-ID: <20160913203619.GC16464@bamboo.slabnet.com> On Tue 2016-Sep-13 13:32:56 -0700, Scott Weeks wrote: > > >--- bryant at backconnect.com wrote: >From: Bryant Townsend > >@ca & Matt - No, we do not plan to ever intentionally perform a >non-authorized BGP hijack in the future. >-------------------------------------------- > > >Bryant, > >Who was the upstream provider? 3223 / VOXILITY https://bgpstream.com/event/54711 >scott -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From surfer at mauigateway.com Tue Sep 13 20:45:06 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Tue, 13 Sep 2016 13:45:06 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160913134506.7605C202@m0087791.ppops.net> --- hugo at slabnet.com wrote: On Tue 2016-Sep-13 13:32:56 -0700, Scott Weeks wrote: >--- bryant at backconnect.com wrote: >From: Bryant Townsend >@ca & Matt - No, we do not plan to ever intentionally perform a >non-authorized BGP hijack in the future. >-------------------------------------------- > >Who was the upstream provider? 3223 / VOXILITY https://bgpstream.com/event/54711 ----------------------------------------------- Oops, I just realized that was mentioned earlier. Apologies for the noise. scott From bryant at backconnect.com Tue Sep 13 21:04:09 2016 From: bryant at backconnect.com (Bryant Townsend) Date: Tue, 13 Sep 2016 14:04:09 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160913203619.GC16464@bamboo.slabnet.com> References: <20160913133256.7605C006@m0087791.ppops.net> <20160913203619.GC16464@bamboo.slabnet.com> Message-ID: @Blake - I think you misinterpreted my remarks of how this experience will shape the future company policy. I meant to portray that the company will not use these tactics again and that any future threats will be handled differently. From hannigan at gmail.com Tue Sep 13 21:05:18 2016 From: hannigan at gmail.com (Martin Hannigan) Date: Tue, 13 Sep 2016 17:05:18 -0400 Subject: Akamai Edgescape query In-Reply-To: References: Message-ID: I do? Drop us a note with your request to peering at akamai.com and we're happy to see if we can accommodate. Best, -M< AS 20940 / AS 32787 On Tue, Sep 13, 2016 at 3:42 AM, Janusz Jezowicz wrote: > Hello, > > We are doing some network research to various cloud/CDNs and we need to run > few tests on Akamai Edgescape. Does anyone has access to run few queries to > geo-locate IPs using Edgescape for us? > > We don't have Akamai contract and it would take us lot of time (and money) > to get one just to run few tests. > > Many thanks for any help! > > Janusz Jezowicz > *Speedchecker Ltd* > *email*: janusz at speedchecker.xyz > *skype*: jezowicz > *phone*: +442032863573 > *web*: www.speedchecker.xyz > The Black Church, St. Mary?s Place, Dublin 7, D07 P4AX, Ireland From owen at delong.com Tue Sep 13 21:15:39 2016 From: owen at delong.com (Owen DeLong) Date: Tue, 13 Sep 2016 17:15:39 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: References: <57D6E8BC.9010404@vaxination.ca> Message-ID: When worded this way in a legal context, I?m pretty sure it is equivalent. That is ?may not? means ?is not allowed to?. Owen > On Sep 13, 2016, at 8:29 AM, Alain Hebert wrote: > > Well "may" is not "must". > > ?260.34. An Internet service provider may not give access to an online > gambling site whose operation is not authorized under Qu?bec law. > > ----- > Alain Hebert ahebert at pubnix.net > PubNIX Inc. > 50 boul. St-Charles > P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 > Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 > > On 09/12/16 13:41, Jean-Francois Mezei wrote: >> As many may know, the province of Qu?bec has passed a law to protect the >> interests of its lottery corporation. >> >> To do so, it will provide ISPs with list of web sites to block (aka: >> only allow its own gambing web site). >> >> There is an opportunity to comment this week in which I will submit. >> >> (I've gathered many arguments over the past little while already). But >> have a specific question today: >> >> Are there examples of an ISP getting sued because it redirected traffic >> that should have gone to original site ? >> >> For instance, user asks for www.google.com and ISP's DNS responds with >> an IP that points to a bing server? >> >> If the risk of a lawsuit is real, then it brings new dimension to >> arguments already made agains that (stupiod) Qu?bec law. >> >> (And it also creates interesting issues for DNS servers from companies >> such as Google which may have a anycast server located in Qu?bec but are >> not considered an ISP and won't receive those documenst from the gov >> with list of websites to block. >> >> From dougm.work at gmail.com Tue Sep 13 19:09:57 2016 From: dougm.work at gmail.com (Doug Montgomery) Date: Tue, 13 Sep 2016 15:09:57 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <68875853-EB91-4BAB-9116-BAF7B28E5117@beckman.org> References: <68875853-EB91-4BAB-9116-BAF7B28E5117@beckman.org> Message-ID: If only there were a global system, with consistent and verifiable security properties, to permit address holders to declare the set of AS's authorized to announce their prefixes, and routers anywhere on the Internet to independently verify the corresponding validity of received announcements. *cough https://www.nanog.org/meetings/abstract?id=2846 cough* I now return us to our discussion of network police, questionnaires for network security, and the use of beer as a motivating force. dougm On Tue, Sep 13, 2016 at 2:51 PM, Mel Beckman wrote: > Blake, > > I concur that these are key questions. Probably _the_ key questions. The > fabric of the Internet is today based on trust, and BGP's integrity is the > core of that trust. > > I realize that BGP hijacking is not uncommon. However, this is the first > time I've seen in it used defensively. I don't see a way to ever bless this > kind of defensive use without compromising that core trust. If Internet > reachability depends on individual providers believing that they are > justified in violating that trust when they are attacked, how can the > Internet stand? > > In addition to the question posed to Bryant about whether he would take > this action again, I would like to add: what about the innocent parties > impacted by your actions? Or do you take the position there were no > innocent parties in the hijacked prefixes? > > -mel via cell > > > On Sep 13, 2016, at 11:40 AM, Blake Hudson wrote: > > > > > > > > Bryant Townsend wrote on 9/13/2016 2:22 AM: > >> This was the point where I decided > >> I needed to go on the offensive to protect myself, my partner, visiting > >> family, and my employees. The actions proved to be extremely effective, > as > >> all forms of harassment and threats from the attackers immediately > stopped. > > > > > > Bryant, what actions, exactly, did you take? This topic seems > intentionally glossed over while you spend a much larger amount of time > explaining the back story and your motivations rather than your actions. > > > > Questions I was left with: > > > > 1. What prefixes have you announced without permission (not just this > > event)? > > 2. How did you identify these prefixes? > > 3. Did you attempt to contact the owner of these prefixes? > > 4. Did you attempt to contact the origin or transit AS of these prefixes? > > 5. What was the process to get your upstream AS to accept these prefix > > announcements? > > 6. Was your upstream AS complicit in allowing you to announce prefixes > > you did not have authorization to announce? > > > -- DougM at Work From cb.list6 at gmail.com Wed Sep 14 00:08:38 2016 From: cb.list6 at gmail.com (Ca By) Date: Tue, 13 Sep 2016 17:08:38 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <68875853-EB91-4BAB-9116-BAF7B28E5117@beckman.org> Message-ID: On Tuesday, September 13, 2016, Doug Montgomery wrote: > If only there were a global system, with consistent and verifiable security > properties, to permit address holders to declare the set of AS's authorized > to announce their prefixes, and routers anywhere on the Internet to > independently verify the corresponding validity of received announcements. > > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* > > I now return us to our discussion of network police, questionnaires for > network security, and the use of beer as a motivating force. > > dougm > > Interesting that backconnect has invalid ROA issued http://bgp.he.net/AS203959#_prefixes On Tue, Sep 13, 2016 at 2:51 PM, Mel Beckman > > wrote: > > > Blake, > > > > I concur that these are key questions. Probably _the_ key questions. The > > fabric of the Internet is today based on trust, and BGP's integrity is > the > > core of that trust. > > > > I realize that BGP hijacking is not uncommon. However, this is the first > > time I've seen in it used defensively. I don't see a way to ever bless > this > > kind of defensive use without compromising that core trust. If Internet > > reachability depends on individual providers believing that they are > > justified in violating that trust when they are attacked, how can the > > Internet stand? > > > > In addition to the question posed to Bryant about whether he would take > > this action again, I would like to add: what about the innocent parties > > impacted by your actions? Or do you take the position there were no > > innocent parties in the hijacked prefixes? > > > > -mel via cell > > > > > On Sep 13, 2016, at 11:40 AM, Blake Hudson > wrote: > > > > > > > > > > > > Bryant Townsend wrote on 9/13/2016 2:22 AM: > > >> This was the point where I decided > > >> I needed to go on the offensive to protect myself, my partner, > visiting > > >> family, and my employees. The actions proved to be extremely > effective, > > as > > >> all forms of harassment and threats from the attackers immediately > > stopped. > > > > > > > > > Bryant, what actions, exactly, did you take? This topic seems > > intentionally glossed over while you spend a much larger amount of time > > explaining the back story and your motivations rather than your actions. > > > > > > Questions I was left with: > > > > > > 1. What prefixes have you announced without permission (not just this > > > event)? > > > 2. How did you identify these prefixes? > > > 3. Did you attempt to contact the owner of these prefixes? > > > 4. Did you attempt to contact the origin or transit AS of these > prefixes? > > > 5. What was the process to get your upstream AS to accept these prefix > > > announcements? > > > 6. Was your upstream AS complicit in allowing you to announce prefixes > > > you did not have authorization to announce? > > > > > > > > > -- > DougM at Work > From hank at efes.iucc.ac.il Wed Sep 14 04:04:13 2016 From: hank at efes.iucc.ac.il (Hank Nussbacher) Date: Wed, 14 Sep 2016 07:04:13 +0300 Subject: "Defensive" BGP hijacking? In-Reply-To: <20f702e2-913a-5c03-8ba2-4513faceb228@ispn.net> References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> <20f702e2-913a-5c03-8ba2-4513faceb228@ispn.net> Message-ID: <82522431-8248-c598-99ea-89995e351eb9@efes.iucc.ac.il> On 13/09/2016 23:22, Blake Hudson wrote: > Ca By wrote on 9/13/2016 2:53 PM: >> On Tuesday, September 13, 2016, Bryant Townsend >> wrote: >> >> Tip to the RIR policy folks, you may want to make this point very >> crisp. A >> BGP ASN is the fundamental accountability control in a inter-domain >> routing. Organizations with repeated offensense need to have their ASN >> revoked, and further there should be controls in places so bad actors >> cannot acquire "burner" ASNs. The RIRs have made it very clear that they will not get involved. Period. -Hank From jfmezei_nanog at vaxination.ca Wed Sep 14 04:14:17 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Wed, 14 Sep 2016 00:14:17 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: References: <57D6E8BC.9010404@vaxination.ca> Message-ID: <57D8CE99.1080101@vaxination.ca> On 2016-09-13 03:42, LHC wrote: > I believe that the CRTC has rules against censorship - meaning that Videotron, Bell etcetera have a choice between following the CRTC code or the provincial law (following one = sanctions from the other), rendering internet service provision to Qu?bec impossible without being a dialup provider from out-of-province. Canada's Telecom Act (*) dates from 1993, which predates the Internet being a primary transporter that drives the economy. The clause being looked at by the CRTC is 36: Content of Messages 36 Except where the Commission approves otherwise, a Canadian carrier shall not control the content or influence the meaning or purpose of telecommunications carried by it for the public. There is not explicit clause about a carrier not modyfying content or blocking access, so one has to frame an issue to fit existing clauses. (For instance, network neutrality is driven mostly by 27(2) (2) No Canadian carrier shall, in relation to the provision of a telecommunications service or the charging of a rate for it, unjustly discriminate or give an undue or unreasonable preference toward any person, including itself, or subject any person to an undue or unreasonable disadvantage. The CRTC asked for supporting evidence to allow it to reach a conclusion that the Qu?bec plan would force ISPs to breach the federal telecommunication law. Because so far, only lawyers have been involved, they have not understood the implications of forcing ISPs to give false DNS answers which goes beyond just blocking packets. (For instance, most ISPs will block packets destined to an external port 25 to reduce spam being emitted by infected customers, but they allow any/all email to be sent via their own servers. There is an element of controlling content but was never challenged. Because Section 36 allows the CRTC to approcve some control of content, it is important to show that the type of control being requested by the QC government show never be allowed because it is far worse than just blocking port 25. (*) http://laws-lois.justice.gc.ca/eng/acts/T-3.4/FullText.html From surfer at mauigateway.com Wed Sep 14 07:09:51 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Wed, 14 Sep 2016 00:09:51 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160914000951.CCA9D8C7@m0086238.ppops.net> --- dougm.work at gmail.com wrote: From: Doug Montgomery If only there were a global system, with consistent and verifiable security properties, to permit address holders to declare the set of AS's authorized to announce their prefixes, and routers anywhere on the Internet to independently verify the corresponding validity of received announcements. *cough https://www.nanog.org/meetings/abstract?id=2846 cough* ------------------------------------------------ Yes, RPKI. That's what I was waiting for. Now we can get to a real discussion... ;-) scott From mel at beckman.org Wed Sep 14 14:51:53 2016 From: mel at beckman.org (Mel Beckman) Date: Wed, 14 Sep 2016 14:51:53 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160914000951.CCA9D8C7@m0086238.ppops.net> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> Scott and Doug, The problem with a new automated enforcement system is that it hobbles both agility and innovation. ISPs have enjoyed simple BGP management, entirely self-regulated, for decades. A global enforcement system, besides being dang hard to do correctly, brings the specter of government interference, since such a system could be overtaken by government entities to manhandle free speech. In my opinion, the community hasn't spent nearly enough time discussing the danger aspect. Being engineers, we focus on technical means, ignoring the fact that we're designing our own guillotine. -mel beckman > On Sep 14, 2016, at 12:10 AM, Scott Weeks wrote: > > > > --- dougm.work at gmail.com wrote: > From: Doug Montgomery > > If only there were a global system, with consistent and verifiable security > properties, to permit address holders to declare the set of AS's authorized > to announce their prefixes, and routers anywhere on the Internet to > independently verify the corresponding validity of received announcements. > > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* > ------------------------------------------------ > > > Yes, RPKI. That's what I was waiting for. Now we can get to > a real discussion... ;-) > > scott From jfmezei_nanog at vaxination.ca Wed Sep 14 19:00:24 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Wed, 14 Sep 2016 15:00:24 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160914000951.CCA9D8C7@m0086238.ppops.net> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: <57D99E48.8080402@vaxination.ca> I got to think about this (dangerous thing :-( Ideally, law enforcement should have the smarts and tools to get involved in DDoS and other similar situations and have the power to compell upstream provider(s) to shut service to a suspect. The current situation appears to be more of a wild-west situation where everyone takes the law into their own hands. It sort of works but everyone knows this lead lead to abuses. If you start to tolerate falsifying BGP, it will likely lead to regular abuses (including intelligence agencies who stad to gain by redirecting traffic to their servers) as well as corporate spies etc. So mechanisms to enforce 0 tolerance are perhaps necessary, even if this means that a few legitimate BGP tricks to save customers from a failing ISP will no longer work. Falsifying BGP can be done by one person without any sanity checks. There is no check for evidence or whether this action is warranted. On the other hand, there is a sanity check if you have to convince an upstream provider to cut access to one of their customers. From Bryan at bryanfields.net Wed Sep 14 20:04:43 2016 From: Bryan at bryanfields.net (Bryan Fields) Date: Wed, 14 Sep 2016 16:04:43 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160914000951.CCA9D8C7@m0086238.ppops.net> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: On 9/14/16 3:09 AM, Scott Weeks wrote: > > Yes, RPKI. That's what I was waiting for. Now we can get to > a real discussion Problem is, RPKI does not work for people with legacy blocks who will not sign a Legacy RSA. ARIN doesn't own or have any say on how we use it, and we're sure as heck not going to sign a legally binding contract saying they do :) I'm a bit ambivalent about BGP hijacking as a DDOS mitigation strategy. Really there is no authority to say it's wrong. If your peers are cool with it, and their peers are cool with it who's to say it's wrong? -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net From morrowc.lists at gmail.com Wed Sep 14 20:59:14 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Wed, 14 Sep 2016 16:59:14 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: On Wed, Sep 14, 2016 at 4:04 PM, Bryan Fields wrote: > On 9/14/16 3:09 AM, Scott Weeks wrote: > > > > Yes, RPKI. That's what I was waiting for. Now we can get to > > a real discussion > > Problem is, RPKI does not work for people with legacy blocks who will not > sign > a Legacy RSA. ARIN doesn't own or have any say on how we use it, and we're > sure it does, move your registration to ripe. (this was also given at nanog or ripe or something, I couldn't remember which was the right one) > sure as heck not going to sign a legally binding contract saying they do :) > > don't have to... see preso. > I'm a bit ambivalent about BGP hijacking as a DDOS mitigation strategy. > Really there is no authority to say it's wrong. If your peers are cool > with > it, and their peers are cool with it who's to say it's wrong? > > -- > Bryan Fields > > 727-409-1194 - Voice > http://bryanfields.net > From surfer at mauigateway.com Wed Sep 14 21:45:09 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Wed, 14 Sep 2016 14:45:09 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160914144509.CCA9ADDC@m0086238.ppops.net> --- jfmezei_nanog at vaxination.ca wrote: From: Jean-Francois Mezei I got to think about this (dangerous thing :-( Ideally, law enforcement should have the smarts and tools to get involved in DDoS and other similar situations and have the power to compell upstream provider(s) to shut service to a suspect. --------------------------------------------------------- Yes, getting law enforcement involved in BGP and network engineering in any way is a dangerous thing. I agree 100%. scott From surfer at mauigateway.com Wed Sep 14 21:48:02 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Wed, 14 Sep 2016 14:48:02 -0700 Subject: "Defensive" BGP hijacking? Message-ID: <20160914144802.CCA9AD4B@m0086238.ppops.net> --- Bryan at bryanfields.net wrote: From: Bryan Fields I'm a bit ambivalent about BGP hijacking as a DDOS mitigation strategy. Really there is no authority to say it's wrong. If your peers are cool with it, and their peers are cool with it who's to say it's wrong? --------------------------------------- Pretty much the entire BGP speaking world. scott From sandy at tislabs.com Wed Sep 14 21:49:55 2016 From: sandy at tislabs.com (Sandra Murphy) Date: Wed, 14 Sep 2016 17:49:55 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <68875853-EB91-4BAB-9116-BAF7B28E5117@beckman.org> Message-ID: <1E3CE598-E6A4-4B70-B72B-2C6DF38BAAE6@tislabs.com> > On Sep 13, 2016, at 8:08 PM, Ca By wrote: > > On Tuesday, September 13, 2016, Doug Montgomery > wrote: > >> If only there were a global system, with consistent and verifiable security >> properties, to permit address holders to declare the set of AS's authorized >> to announce their prefixes, and routers anywhere on the Internet to >> independently verify the corresponding validity of received announcements. >> >> *cough https://www.nanog.org/meetings/abstract?id=2846 cough* >> >> I now return us to our discussion of network police, questionnaires for >> network security, and the use of beer as a motivating force. >> >> dougm >> >> > Interesting that backconnect has invalid ROA issued > > http://bgp.he.net/AS203959#_prefixes Well, no, that?s not what the bgp.he.net (live long and prosper!) icons mean. There is nothing invalid about the ROA. And BackConnect did not issue it. The red key means that AS203959 BackConnect Security LLC is announcing routes for 181.215.239.0/24, 191.96.128.0/24. etc that are invalid according to the RPKI. Someone else with the right to use 181.215.239.0/24, 191.96.128.0/24. etc, created ROAs authorizing some other AS(s) to originate the route. You can look at the ROAs for those prefixes with red keys in http://rpki-browser.realmv6.org (and with the RPKIviz and EOM tools from www.securerouting.net). You can see that there are ROAs for 181.214.0.0/15 for AS50673, AS60458, AS61440. and ROAs for 191.96.0.0/16 for AS61440, AS60458, AS29073, AS33182, AS37692. and ROAs for 191.101.0.0/16 for AS60458, AS37692, AS61440. Except. It looks like, maybe, BackConnect was re-allocated the four prefixes with red keys, and whoever reallocated the prefixes to them created ROAs for their aggregate ? but not for their customers. See for example http://bgp.he.net/net/191.96.128.0/24 (and http://bgp.he.net/net/191.101.191.0/24) This can occurred as the world backfills the existing allocations and the customers don?t keep up and their providers aren?t careful. Some RPKI implementations will warn you that the ROA you are about to create will make existing routes appear invalid to the RPKI. Just for fun, take a look at the IRR entries for the four prefixes with red key icons: route: 191.96.128.0/24 descr: Clan Servers origin: AS20473 notify: net-support at ap.equinix.com notify: noc at ap.equinix.com mnt-by: MAINT-EQUINIXPAC changed: mvillalobos at ap.equinix.com 20151229 source: NTTCOM route: 191.96.129.0/24 descr: Clan Servers origin: AS20473 notify: net-support at ap.equinix.com notify: noc at ap.equinix.com mnt-by: MAINT-EQUINIXPAC changed: mvillalobos at ap.equinix.com 20151229 source: NTTCOM route: 191.101.191.0/24 descr: Clan Servers origin: AS20473 notify: net-support at ap.equinix.com notify: ap-noc at ap.equinix.com mnt-by: MAINT-EQUINIXPAC changed: mporquez at ap.equinix.com 20151227 source: NTTCOM route: 181.215.239.0/24 descr: Proxy route object registered by AS2764 origin: AS45177 remarks: This route object was created by AAPT on behalf of a customer. remarks: As some of AAPTs upstream networks filter based on IRR objects, remarks: this route object has been created to ensure that the advertisement remarks: of this prefix is not rejected. notify: routing.shared at aapt.com.au mnt-by: MAINT-AS2764 changed: nobody at aapt.com.au 20160106 source: RADB (like the ?nobody? part???) At least the aggregate announcements aren?t proxies. route: 181.214.0.0/19 descr: Digital Energy Technologies LTD. origin: AS61440 mnt-by: MAINT-AS61440 changed: modestas at host1plus.com 20140814 #13:04:53Z source: RADB route: 191.101.0.0/21 descr: Digital Energy Technologies LTD. origin: AS25761 mnt-by: MAINT-AS61440 changed: modestas at host1plus.com 20150610 #08:53:38Z source: RADB ?Sandy (Since bgp.he.net changes as the routing world changes, the red keys could be gone by the time anyone looks, of course.) > > On Tue, Sep 13, 2016 at 2:51 PM, Mel Beckman > >> wrote: >> >>> Blake, >>> >>> I concur that these are key questions. Probably _the_ key questions. The >>> fabric of the Internet is today based on trust, and BGP's integrity is >> the >>> core of that trust. >>> >>> I realize that BGP hijacking is not uncommon. However, this is the first >>> time I've seen in it used defensively. I don't see a way to ever bless >> this >>> kind of defensive use without compromising that core trust. If Internet >>> reachability depends on individual providers believing that they are >>> justified in violating that trust when they are attacked, how can the >>> Internet stand? >>> >>> In addition to the question posed to Bryant about whether he would take >>> this action again, I would like to add: what about the innocent parties >>> impacted by your actions? Or do you take the position there were no >>> innocent parties in the hijacked prefixes? >>> >>> -mel via cell >>> >>>> On Sep 13, 2016, at 11:40 AM, Blake Hudson > > wrote: >>>> >>>> >>>> >>>> Bryant Townsend wrote on 9/13/2016 2:22 AM: >>>>> This was the point where I decided >>>>> I needed to go on the offensive to protect myself, my partner, >> visiting >>>>> family, and my employees. The actions proved to be extremely >> effective, >>> as >>>>> all forms of harassment and threats from the attackers immediately >>> stopped. >>>> >>>> >>>> Bryant, what actions, exactly, did you take? This topic seems >>> intentionally glossed over while you spend a much larger amount of time >>> explaining the back story and your motivations rather than your actions. >>>> >>>> Questions I was left with: >>>> >>>> 1. What prefixes have you announced without permission (not just this >>>> event)? >>>> 2. How did you identify these prefixes? >>>> 3. Did you attempt to contact the owner of these prefixes? >>>> 4. Did you attempt to contact the origin or transit AS of these >> prefixes? >>>> 5. What was the process to get your upstream AS to accept these prefix >>>> announcements? >>>> 6. Was your upstream AS complicit in allowing you to announce prefixes >>>> you did not have authorization to announce? >>>> >>> >> >> >> >> -- >> DougM at Work >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mel at beckman.org Wed Sep 14 22:04:26 2016 From: mel at beckman.org (Mel Beckman) Date: Wed, 14 Sep 2016 22:04:26 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org>, Message-ID: <2AB19C93-0E35-4AA0-B781-EA4653A729A8@beckman.org> Doug, I was basing my comments on your statement "If only there were a global system.." However you slice or dice it, the tyranny implications have not yet been addressed. That certainly needs to be in front of any technical idea such as RPKI. Although I haven't participated in the OT&E, nothing I've read in RFC 6810 talks about these issues. It talks about authentication and transport security, but doesn't talk about the potential for government interference. -mel beckman On Sep 14, 2016, at 8:22 AM, Doug Montgomery > wrote: Mel, If you are speaking of RPKI based origin validation, I am not sure "automated / global enforcement system" is a useful description. It does provide a consistent means for address holders to declare AS's authorized to announce prefixes, and a means for remote ASs to compare received updates vs such declarations. What the receiving AS does with the validation information is strictly a local policy matter. Frankly, this is no more a "new automated enforcement system" than IRR-based route filtering has been for 20 years. The only difference is that there is a consistent security model across all 5 RIRs as to who can make such declarations and it is tightly tied to the address allocation business process. I have seen a lot of FUD about the specter of interference, but not a lot of serious thought / discussion. Having a serious technical discussion of potential risks and mitigations in the system would be useful. dougm On Wed, Sep 14, 2016 at 10:51 AM, Mel Beckman > wrote: Scott and Doug, The problem with a new automated enforcement system is that it hobbles both agility and innovation. ISPs have enjoyed simple BGP management, entirely self-regulated, for decades. A global enforcement system, besides being dang hard to do correctly, brings the specter of government interference, since such a system could be overtaken by government entities to manhandle free speech. In my opinion, the community hasn't spent nearly enough time discussing the danger aspect. Being engineers, we focus on technical means, ignoring the fact that we're designing our own guillotine. -mel beckman > On Sep 14, 2016, at 12:10 AM, Scott Weeks > wrote: > > > > --- dougm.work at gmail.com wrote: > From: Doug Montgomery > > > If only there were a global system, with consistent and verifiable security > properties, to permit address holders to declare the set of AS's authorized > to announce their prefixes, and routers anywhere on the Internet to > independently verify the corresponding validity of received announcements. > > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* > ------------------------------------------------ > > > Yes, RPKI. That's what I was waiting for. Now we can get to > a real discussion... ;-) > > scott -- DougM at Work From rsk at gsp.org Wed Sep 14 22:34:09 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Wed, 14 Sep 2016 18:34:09 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: <20160914223409.GA8864@gsp.org> On Wed, Sep 14, 2016 at 04:04:43PM -0400, Bryan Fields wrote: > I'm a bit ambivalent about BGP hijacking as a DDOS mitigation strategy. > Really there is no authority to say it's wrong. If your peers are cool with > it, and their peers are cool with it who's to say it's wrong? Meeting abuse with abuse never works out. It's tempting (and even trendy these days in portions of the security world which advocate striking back at putative attackers, never mind that attack attribution is almost entirely an unsolved problem in computing). It's emotionally satisfying. It's sometimes momentarily effective. But all it really does it open up still more attack vectors and accelerate the spiral to the bottom. Object lesson: Verizon's deployment of SAV as an alleged anti-spam measure ~15 years ago. It didn't take long for attackers to figure out how to leverage it to their advantage, which of course they did. So don't do it. It may take 5 minutes or 5 years, but it will eventually become apparent that it's a really bad idea. And when it does, you won't be able to get those 5 minutes or 5 years back, nor will you be able to undo the damage. ---rsk From woody at pch.net Thu Sep 15 00:04:17 2016 From: woody at pch.net (Bill Woodcock) Date: Wed, 14 Sep 2016 17:04:17 -0700 Subject: PCH peering survey 2016 Message-ID: Background: Five years ago PCH conducted the first, and to date only, comprehensive survey characterizing Internet peering agreements. The document that resulted can be found here: https://www.pch.net/resources/Papers/peering-survey/PCH-Peering-Survey-2011.pdf That document was one of the principal inputs to an important document that the OECD publishes every five years, one that recommends communications regulatory policy to OECD member nations: http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=DSTI/ICCP/CISP(2011)2/FINAL&docLanguage=En The survey had several useful findings which hadn?t previously been established as fact?most notably the portion of peering relationships that are ?handshake? agreements, without written contract. These findings have improved the regulatory environments in which many of us operate our networks. At the time of the 2011 survey, we committed to repeating the survey every five years, so as to provide an ongoing indication of the direction peering trends take. It?s now five years later, so we?re repeating the survey. The survey is global in scope, and our goal is to reflect the diversity of peering agreements in the world; we?re interested in large ISPs and small ISPs, ISPs in Afghanistan and in Zimbabwe, bilateral agreements and multilateral, private and public. Our intent is to be as comprehensive as possible. In 2011, the responses we received represented 86% of all of the world?s ISPs and 96 countries. We would like to be at least as inclusive this time. Privacy: In 2011, we promised to collect the smallest set of data necessary to answer the questions, to perform the analysis immediately, and not to retain the data after the analysis was accomplished. In that way, we ensured that the privacy of respondents was fully protected. We did as we said, no data was leaked, and the whole community benefited from the trust that was extended to us. We ask for your trust again now as we make the same commitment to protect the privacy of all respondents, using the same process as last time. We are asking for no more data than is absolutely necessary. We will perform the analysis immediately upon receiving all of the data. We will delete the data once the analysis has been performed. The Survey: We would like to know the following five pieces of information relative to each Autonomous System you peer with: ? Your ASN ? Your peer?s ASN (peers only, not upstream transit providers or downstream customers) ? Whether a written and signed peering agreement exists (the alternative being a less formal arrangement, such as a "handshake agreement") ? Whether the terms are roughly symmetric (the alternative being that they describe an agreement with different terms for each of the two parties, such as one compensating the other, or one receiving more or fewer than full customer routes) ? Whether a jurisdiction of governing law is defined ? Whether IPv6 routes are being exchanged (this year, we?ll still assume that IPv4 are) The easiest way for us to receive the information is as a tab-text or CSV file or an Excel spreadsheet, consisting of rows with the following columns: Your ASN: Integer Peer ASN: Integer Written agreement: Boolean Symmetric: Boolean Governing Law: ISO 3166 two-digit country-code, or empty IPv6 Routes: Boolean For instance: 42 715 false true us true 42 3856 true true us true We are asking for the ASNs only so we can avoid double-counting a single pair of peers when we hear from both of them, and so that when we hear about a relationship in responses from both peers we can see how closely the two responses match, an important check on the quality of the survey. As soon as we've collated the data, we'll strip the ASNs to protect privacy, and only the final aggregate statistics will be published. We will never disclose any ASN or any information about any ASN. We already have more than 8,000 ASN-pair relationships documented, and we hope to receive as many more as possible. We'd like to finish collecting data by the end of September, about two weeks from now. If you?re peering with an MLPA route-server, you?re welcome to include just the route-server?s ASN, if that?s easiest, rather than trying to include each of the peer ASNs on the other side of the route-server. Either way is fine. If all of your sessions have the same characteristics, you can just tell us what those characteristics are once, your own ASN once, and give us a simple list of your peer ASNs. If your number of peers is small enough to be pasted or typed into an email, rather than attached as a file, and that?s simpler, just go ahead and do that. If you have written peering agreements that are covered by non-disclosure agreements, or if your organizational policy precludes disclosing your peers, but you?d still like to participate in the survey, please let us know, and we?ll work with whatever information you?re able to give us and try to ensure that your practices are statistically represented in our results. If you're able to help us, please email me the data in whatever form you can. If you need a non-disclosure, we're happy to sign one. Finally, if there are any other questions you?d like to see answered in the future, please let us know so that we can consider addressing them in the 2021 survey. The question about IPv6 routing in this year?s survey is there because quite a few of the 2011 respondents asked us to include it this time. Please respond by replying to this email, before the end of September. Thank you for considering participating. We very much appreciate it, and we look forward to returning the results to the community. -Bill Woodcock Executive Director Packet Clearing House -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From marka at isc.org Thu Sep 15 00:15:16 2016 From: marka at isc.org (Mark Andrews) Date: Thu, 15 Sep 2016 10:15:16 +1000 Subject: Can someone from Amazon please answer. In-Reply-To: Your message of "Wed, 24 Aug 2016 09:37:10 +1000." <20160823233710.8DC3A5206AD7@rock.dv.isc.org> References: <20160823233710.8DC3A5206AD7@rock.dv.isc.org> Message-ID: <20160915001516.B42215449756@rock.dv.isc.org> In message <20160823233710.8DC3A5206AD7 at rock.dv.isc.org>, Mark Andrews writes: > > I'm curious. What are you trying to achieve by blocking EDNS version > negotiation? Is it really too hard to return BADVERS to a EDNS > query with version != 0 along with the version of EDNS you support > in the version field? Are you deliberately trying to prevent the > IETF from deciding to bump the EDNS version in the future? Do you > have firewalls that have this behaviour hard coded? Do you even > test for RFC compliance? > > Mark > > lostoncampus.com.au. @205.251.195.156 (ns-924.awsdns-51.net.): dns=ok edns=ok > edns1=timeout edns at 512=ok ednsopt=ok edns1opt=timeout do=ok ednsflags=ok opt > list=ok,nsid,subnet signed=ok ednstcp=ok > lostoncampus.com.au. @205.251.192.78 (ns-78.awsdns-09.com.): dns=ok edns=ok e > dns1=timeout edns at 512=ok ednsopt=ok edns1opt=timeout do=ok ednsflags=ok optli > st=ok,nsid,subnet signed=ok ednstcp=ok > lostoncampus.com.au. @205.251.196.198 (ns-1222.awsdns-24.org.): dns=ok edns=o > k edns1=timeout edns at 512=ok ednsopt=ok edns1opt=timeout do=ok ednsflags=ok op > tlist=ok,nsid,subnet signed=ok ednstcp=ok > lostoncampus.com.au. @205.251.199.20 (ns-1812.awsdns-34.co.uk.): dns=ok edns= > ok edns1=timeout edns at 512=ok ednsopt=ok edns1opt=timeout do=ok ednsflags=ok o > ptlist=ok,nsid,subnet signed=ok ednstcp=ok > > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org Amazon are updating their servers/firewalls so they no longer timeout. They still need to return a EDNS response but it is a step in the right direction. Thanks for improving the situation. It makes for some dramatic changes in the EDNS(1) and EDNS(1) + Unknown EDNS option failure mode and response graphs at https://ednscomp.isc.org/compliance/summary.html Mark % dig soa lostoncampus.com.au @205.251.195.156 +edns=1 +noednsneg +norec ; <<>> DiG 9.11.0rc1 <<>> soa lostoncampus.com.au @205.251.195.156 +edns=1 +noednsneg +norec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52640 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0 ;; QUESTION SECTION: ;lostoncampus.com.au. IN SOA ;; ANSWER SECTION: lostoncampus.com.au. 900 IN SOA ns-1222.awsdns-24.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; AUTHORITY SECTION: lostoncampus.com.au. 172800 IN NS ns-1222.awsdns-24.org. lostoncampus.com.au. 172800 IN NS ns-1812.awsdns-34.co.uk. lostoncampus.com.au. 172800 IN NS ns-78.awsdns-09.com. lostoncampus.com.au. 172800 IN NS ns-924.awsdns-51.net. ;; Query time: 132 msec ;; SERVER: 205.251.195.156#53(205.251.195.156) ;; WHEN: Thu Sep 15 10:09:42 EST 2016 ;; MSG SIZE rcvd: 237 % Checking: 'lostoncampus.com.au' as at 2016-09-15T00:07:37Z lostoncampus.com.au @205.251.196.198 (ns-1222.awsdns-24.org.): dns=ok edns=ok edns1=status,noopt,soa edns at 512=ok ednsopt=ok edns1opt=status,noopt,soa do=ok ednsflags=ok edns at 512tcp=ok optlist=nsid,subnet lostoncampus.com.au @205.251.199.20 (ns-1812.awsdns-34.co.uk.): dns=ok edns=ok edns1=status,noopt,soa edns at 512=ok ednsopt=ok edns1opt=status,noopt,soa do=ok ednsflags=ok edns at 512tcp=ok optlist=nsid,subnet lostoncampus.com.au @205.251.192.78 (ns-78.awsdns-09.com.): dns=ok edns=ok edns1=timeout edns at 512=ok ednsopt=ok edns1opt=timeout do=ok ednsflags=ok edns at 512tcp=ok optlist=nsid,subnet lostoncampus.com.au @205.251.195.156 (ns-924.awsdns-51.net.): dns=ok edns=ok edns1=status,noopt,soa edns at 512=ok ednsopt=ok edns1opt=status,noopt,soa do=ok ednsflags=ok edns at 512tcp=ok optlist=nsid,subnet The Following Tests Failed EDNS - Unknown Version Handling (edns1) dig +nocookie +norec +noad +edns=1 +noednsneg soa zone @server expect: BADVERS expect: OPT record with version set to 0 expect: not to see SOA See RFC6891, 6.1.3. OPT Record TTL Field Use EDNS - Unknown Version with Unknown Option Handling (edns1opt) dig +nocookie +norec +noad +edns=1 +noednsneg +ednsopt=100 soa zone @server expect: BADVERS expect: OPT record with version set to 0 expect: not to see SOA expect: that the option will not be present in response See RFC6891 Codes ok - test passed. nsid - NSID supported. subnet - EDNS Client Subnet supported. soa - SOA record found when not expected. noopt - OPT record not found when expected. status - expected rcode status code not found. timeout - lookup timed out. To retrieve this report in the future: https://ednscomp.isc.org/ednscomp/0e5c781801 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From marka at isc.org Thu Sep 15 07:31:34 2016 From: marka at isc.org (Mark Andrews) Date: Thu, 15 Sep 2016 17:31:34 +1000 Subject: QWEST.NET can you fix your nameservers Message-ID: <20160915073135.2670C54534F2@rock.dv.isc.org> In case anyone is wondering why I've been harping on about EDNS compliance this is why. Failure to follow the protocol can result in DNS lookup failures. nara.gov is signed and the recursive server performs DNSSEC validation and sends queries with DNS COOKIEs. BADVERS is NOT a valid response to a EDNS version 0 query. Can you please contact your DNS vendor for a fix. QWEST isn't the only DNS provider that has broken nameservers. One shouldn't have to try and contact every DNS operator to get them to use protocol compliant servers. Mark ;; BADCOOKIE, retrying. ; <<>> DiG 9.11.0rc1 <<>> nara.gov ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 5744 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 85faf1e39a1a6a149bebd00a57da4b266b8546c1b75015db (good) ;; QUESTION SECTION: ;nara.gov. IN A ;; Query time: 5000 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Thu Sep 15 17:17:58 EST 2016 ;; MSG SIZE rcvd: 65 Checking: 'nara.gov' as at 2016-09-15T07:16:32Z nara.gov @63.150.72.5 (sauthns1.qwest.net.): dns=ok edns=ok edns1=ok edns at 512=ok ednsopt=badvers,nosoa edns1opt=ok do=nodo ednsflags=ok edns at 512tcp=ok optlist=badvers,nosoa nara.gov @2001:428::7 (sauthns1.qwest.net.): dns=ok edns=ok edns1=ok edns at 512=ok ednsopt=badvers,nosoa edns1opt=ok do=nodo ednsflags=ok edns at 512tcp=ok optlist=badvers,nosoa nara.gov @208.44.130.121 (sauthns2.qwest.net.): dns=ok edns=ok edns1=ok edns at 512=ok ednsopt=badvers,nosoa edns1opt=ok do=nodo ednsflags=ok edns at 512tcp=ok optlist=badvers,nosoa nara.gov @2001:428::8 (sauthns2.qwest.net.): dns=ok edns=ok edns1=ok edns at 512=ok ednsopt=badvers,nosoa edns1opt=ok do=nodo ednsflags=ok edns at 512tcp=ok optlist=badvers,nosoa The Following Tests Failed EDNS - Unknown Option Handling (ednsopt) dig +nocookie +norec +noad +ednsopt=100 soa zone @server expect: SOA expect: NOERROR expect: OPT record with version set to 0 expect: that the option will not be present in response See RFC6891, 6.1.2 Wire Format EDNS - DO=1 (do) dig +nocookie +norec +noad +dnssec soa zone @server expect: SOA expect: NOERROR expect: OPT record with version set to 0 expect: DO flag in response if RRSIG is present in response See RFC3225 EDNS - Supported Options Probe (optlist) dig +edns +noad +norec +nsid +subnet=0.0.0.0/0 +expire +cookie -q zone @server expect: NOERROR expect: OPT record with version set to 0 See RFC6891 Codes ok - test passed. nodo - EDNS DO flag not echoed. nosoa - SOA record not found when expected. badvers - BADVERS returned. To retrieve this report in the future: https://ednscomp.isc.org/ednscomp/25f2ebe619 -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From aaron at heyaaron.com Thu Sep 15 16:22:10 2016 From: aaron at heyaaron.com (Aaron C. de Bruyn) Date: Thu, 15 Sep 2016 09:22:10 -0700 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <20160915073135.2670C54534F2@rock.dv.isc.org> References: <20160915073135.2670C54534F2@rock.dv.isc.org> Message-ID: On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews wrote: > QWEST isn't the only DNS provider that has broken nameservers. One > shouldn't have to try and contact every DNS operator to get them to > use protocol compliant servers. > Save yourself some time. Contact the DNS software vendors. ;) -A From bill at herrin.us Thu Sep 15 16:43:57 2016 From: bill at herrin.us (William Herrin) Date: Thu, 15 Sep 2016 12:43:57 -0400 Subject: QWEST.NET can you fix your nameservers In-Reply-To: References: <20160915073135.2670C54534F2@rock.dv.isc.org> Message-ID: On Thu, Sep 15, 2016 at 12:22 PM, Aaron C. de Bruyn wrote: > On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews wrote: >> QWEST isn't the only DNS provider that has broken nameservers. One >> shouldn't have to try and contact every DNS operator to get them to >> use protocol compliant servers. > > Save yourself some time. Contact the DNS software vendors. ;) I'd bet he already has. This looks like a name-and-shame to me, and probably deserved. -Bill -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From Valdis.Kletnieks at vt.edu Thu Sep 15 17:19:24 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Thu, 15 Sep 2016 13:19:24 -0400 Subject: QWEST.NET can you fix your nameservers In-Reply-To: References: <20160915073135.2670C54534F2@rock.dv.isc.org> Message-ID: <13615.1473959964@turing-police.cc.vt.edu> On Thu, 15 Sep 2016 09:22:10 -0700, "Aaron C. de Bruyn" said: > On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews wrote: > > > QWEST isn't the only DNS provider that has broken nameservers. One > > shouldn't have to try and contact every DNS operator to get them to > > use protocol compliant servers. > Save yourself some time. Contact the DNS software vendors. ;) Often, the vendor *is* on the ball, but the customer isn't. Remember that Windows XP didn't enable IPv6 by default, and *still* has some 10% market share. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 830 bytes Desc: not available URL: From jason+nanog at lixfeld.ca Thu Sep 15 18:07:01 2016 From: jason+nanog at lixfeld.ca (Jason Lixfeld) Date: Thu, 15 Sep 2016 14:07:01 -0400 Subject: Any ISPs using AS852 for IP Transit? Message-ID: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be interested in hearing from you. I?d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we?re the only ones? Thanks in advance! From math at sizone.org Thu Sep 15 18:28:50 2016 From: math at sizone.org (Ken Chase) Date: Thu, 15 Sep 2016 14:28:50 -0400 Subject: charges for prefix filter updates (was Re: Any ISPs using AS852 for IP Transit?) In-Reply-To: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> Message-ID: <20160915182850.GD15995@sizone.org> I feel this can be a public topic: Rogers just charged us that for an update (one update, multiple entries). We had to go through their quotation machinery too, took like 4-5 days. Additional time was wasted because we contacted their tech dept directly at the start. (which is what I do for all my other upstreams...) Kinda brutal. Cogent and HE nor NAC or Yipes or Tata ever did that to us. Nickle and diming -- why, cuz transit is a cheap commodity now, gotta make the cash somewhere? That said Cogent offered us a static /26 along side our BGP years ago then warned us it'd be $50/mo or something for that # of ips going forward. We didnt need it so dispensed with it. /kc On Thu, Sep 15, 2016 at 02:07:01PM -0400, Jason Lixfeld said: >If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I???d be interested in hearing from you. > >I???d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we???re the only ones??? > >Thanks in advance! -- Ken Chase - math at sizone.org Toronto Canada From joelja at bogus.com Thu Sep 15 18:48:05 2016 From: joelja at bogus.com (joel jaeggli) Date: Thu, 15 Sep 2016 11:48:05 -0700 Subject: charges for prefix filter updates (was Re: Any ISPs using AS852 for IP Transit?) In-Reply-To: <20160915182850.GD15995@sizone.org> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <20160915182850.GD15995@sizone.org> Message-ID: <207f3100-698d-c602-370c-e63e5e3ba987@bogus.com> On 9/15/16 11:28 AM, Ken Chase wrote: > I feel this can be a public topic: > > Rogers just charged us that for an update (one update, multiple entries). > We had to go through their quotation machinery too, took like 4-5 days. Additional > time was wasted because we contacted their tech dept directly at the start. (which > is what I do for all my other upstreams...) > > Kinda brutal. Coordination problems are a point of high friction and cost for low margin products. I generally prefer that my providers be able to generate prefix filters on the basis of route objects, If it's not part of their service offering; how costs are assigned for service requests is going to be part of contract negotiions. joel > Cogent and HE nor NAC or Yipes or Tata ever did that to us. > > Nickle and diming -- why, cuz transit is a cheap commodity now, gotta make the > cash somewhere? > > That said Cogent offered us a static /26 along side our BGP years ago then warned > us it'd be $50/mo or something for that # of ips going forward. We didnt need it > so dispensed with it. > > /kc > > > On Thu, Sep 15, 2016 at 02:07:01PM -0400, Jason Lixfeld said: > >If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I???d be interested in hearing from you. > > > >I???d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we???re the only ones??? > > > >Thanks in advance! > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: OpenPGP digital signature URL: From schecter at gmail.com Thu Sep 15 18:50:21 2016 From: schecter at gmail.com (Steven Schecter) Date: Thu, 15 Sep 2016 14:50:21 -0400 Subject: Any ISPs using AS852 for IP Transit? In-Reply-To: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> Message-ID: I question their motivation here and would follow up by asking if they support filtering by IRRdb and are merely trying to encourage the practice? /Steve On Thu, Sep 15, 2016 at 2:07 PM, Jason Lixfeld wrote: > If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be > interested in hearing from you. > > I?d like to compare notes to see if you are also paying $250 for each BGP > prefix filter updated request, or if we?re the only ones? > > Thanks in advance! -- Steven J. Schecter (m) 917.676.1646 From aaron at heyaaron.com Thu Sep 15 18:55:03 2016 From: aaron at heyaaron.com (Aaron C. de Bruyn) Date: Thu, 15 Sep 2016 11:55:03 -0700 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <13615.1473959964@turing-police.cc.vt.edu> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <13615.1473959964@turing-police.cc.vt.edu> Message-ID: On Thu, Sep 15, 2016 at 10:19 AM, wrote: > Remember that Windows XP didn't enable IPv6 by default, and *still* has > some 10% > market share. > Yeah, I'm still fighting that battle. https://goo.gl/photos/xFguK4FL2iydnLhE7 -A From jason+nanog at lixfeld.ca Thu Sep 15 19:09:33 2016 From: jason+nanog at lixfeld.ca (Jason Lixfeld) Date: Thu, 15 Sep 2016 15:09:33 -0400 Subject: Any ISPs using AS852 for IP Transit? In-Reply-To: References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> Message-ID: <9B17C89B-5FCB-4030-A12A-48FECAF2D090@lixfeld.ca> Last time I asked, that wasn?t something that they had implemented, and had no definite plans to do so within any timeframe that was on their radar. > On Sep 15, 2016, at 2:50 PM, Steven Schecter wrote: > > I question their motivation here and would follow up by asking if they support filtering by IRRdb and are merely trying to encourage the practice? > > > /Steve > > On Thu, Sep 15, 2016 at 2:07 PM, Jason Lixfeld wrote: > If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be interested in hearing from you. > > I?d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we?re the only ones? > > Thanks in advance! > > > > -- > Steven J. Schecter > (m) 917.676.1646 From hugo at slabnet.com Thu Sep 15 19:15:29 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Thu, 15 Sep 2016 12:15:29 -0700 Subject: Any ISPs using AS852 for IP Transit? In-Reply-To: <9B17C89B-5FCB-4030-A12A-48FECAF2D090@lixfeld.ca> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <9B17C89B-5FCB-4030-A12A-48FECAF2D090@lixfeld.ca> Message-ID: <20160915191529.GE16464@bamboo.slabnet.com> So, to be blunt, I would cast this as their charging you NRC for manual work because of their failure to automate this. -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal On Thu 2016-Sep-15 15:09:33 -0400, Jason Lixfeld wrote: >Last time I asked, that wasn?t something that they had implemented, and had no definite plans to do so within any timeframe that was on their radar. > >> On Sep 15, 2016, at 2:50 PM, Steven Schecter wrote: >> >> I question their motivation here and would follow up by asking if they support filtering by IRRdb and are merely trying to encourage the practice? >> >> >> /Steve >> >> On Thu, Sep 15, 2016 at 2:07 PM, Jason Lixfeld wrote: >> If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be interested in hearing from you. >> >> I?d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we?re the only ones? >> >> Thanks in advance! >> >> >> >> -- >> Steven J. Schecter >> (m) 917.676.1646 > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From dougm.work at gmail.com Wed Sep 14 15:21:41 2016 From: dougm.work at gmail.com (Doug Montgomery) Date: Wed, 14 Sep 2016 11:21:41 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> Message-ID: Mel, If you are speaking of RPKI based origin validation, I am not sure "automated / global enforcement system" is a useful description. It does provide a consistent means for address holders to declare AS's authorized to announce prefixes, and a means for remote ASs to compare received updates vs such declarations. What the receiving AS does with the validation information is strictly a local policy matter. Frankly, this is no more a "new automated enforcement system" than IRR-based route filtering has been for 20 years. The only difference is that there is a consistent security model across all 5 RIRs as to who can make such declarations and it is tightly tied to the address allocation business process. I have seen a lot of FUD about the specter of interference, but not a lot of serious thought / discussion. Having a serious technical discussion of potential risks and mitigations in the system would be useful. dougm On Wed, Sep 14, 2016 at 10:51 AM, Mel Beckman wrote: > Scott and Doug, > > The problem with a new automated enforcement system is that it hobbles > both agility and innovation. ISPs have enjoyed simple BGP management, > entirely self-regulated, for decades. A global enforcement system, besides > being dang hard to do correctly, brings the specter of government > interference, since such a system could be overtaken by government entities > to manhandle free speech. > > In my opinion, the community hasn't spent nearly enough time discussing > the danger aspect. Being engineers, we focus on technical means, ignoring > the fact that we're designing our own guillotine. > > -mel beckman > > > On Sep 14, 2016, at 12:10 AM, Scott Weeks > wrote: > > > > > > > > --- dougm.work at gmail.com wrote: > > From: Doug Montgomery > > > > If only there were a global system, with consistent and verifiable > security > > properties, to permit address holders to declare the set of AS's > authorized > > to announce their prefixes, and routers anywhere on the Internet to > > independently verify the corresponding validity of received > announcements. > > > > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* > > ------------------------------------------------ > > > > > > Yes, RPKI. That's what I was waiting for. Now we can get to > > a real discussion... ;-) > > > > scott > -- DougM at Work From jason+nanog at lixfeld.ca Thu Sep 15 19:21:52 2016 From: jason+nanog at lixfeld.ca (Jason Lixfeld) Date: Thu, 15 Sep 2016 15:21:52 -0400 Subject: Any ISPs using AS852 for IP Transit? In-Reply-To: <20160915191529.GE16464@bamboo.slabnet.com> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <9B17C89B-5FCB-4030-A12A-48FECAF2D090@lixfeld.ca> <20160915191529.GE16464@bamboo.slabnet.com> Message-ID: <93F8A4A5-B3A3-47C0-A4AC-C630F1D52278@lixfeld.ca> Sure. My question was whether every TELUS BGP customer was being charged for these too, or if I?m the only one. If I?m the only one, then I?m obviously caught in some administrative black hole there that I would like to get myself out of. This is something that has only started happening in the last 6 months or so. Prior to that, we were never charged by them for these requests. Unfortunately, my sales rep has been less than helpful in trying to understand what changed to make us susceptible to these new charges. > On Sep 15, 2016, at 3:15 PM, Hugo Slabbert wrote: > > So, to be blunt, I would cast this as their charging you NRC for manual work because of their failure to automate this. > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal > > On Thu 2016-Sep-15 15:09:33 -0400, Jason Lixfeld wrote: > >> Last time I asked, that wasn?t something that they had implemented, and had no definite plans to do so within any timeframe that was on their radar. >> >>> On Sep 15, 2016, at 2:50 PM, Steven Schecter wrote: >>> >>> I question their motivation here and would follow up by asking if they support filtering by IRRdb and are merely trying to encourage the practice? >>> >>> >>> /Steve >>> >>> On Thu, Sep 15, 2016 at 2:07 PM, Jason Lixfeld wrote: >>> If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be interested in hearing from you. >>> >>> I?d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we?re the only ones? >>> >>> Thanks in advance! >>> >>> >>> >>> -- >>> Steven J. Schecter >>> (m) 917.676.1646 >> From owen at delong.com Thu Sep 15 20:03:44 2016 From: owen at delong.com (Owen DeLong) Date: Thu, 15 Sep 2016 16:03:44 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <57D8CE99.1080101@vaxination.ca> References: <57D6E8BC.9010404@vaxination.ca> <57D8CE99.1080101@vaxination.ca> Message-ID: <7B8AC8E8-83C2-4BD9-AA34-4212DF995822@delong.com> > On Sep 14, 2016, at 12:14 AM, Jean-Francois Mezei wrote: > > On 2016-09-13 03:42, LHC wrote: >> I believe that the CRTC has rules against censorship - meaning that Videotron, Bell etcetera have a choice between following the CRTC code or the provincial law (following one = sanctions from the other), rendering internet service provision to Qu?bec impossible without being a dialup provider from out-of-province. > > > Canada's Telecom Act (*) dates from 1993, which predates the Internet > being a primary transporter that drives the economy. > > The clause being looked at by the CRTC is 36: > > Content of Messages > > 36 Except where the Commission approves otherwise, a Canadian carrier > shall not control the content or influence the meaning or purpose of > telecommunications carried by it for the public. > > There is not explicit clause about a carrier not modyfying content or > blocking access, so one has to frame an issue to fit existing clauses. Please explain to me how one modifies a request or response without managing to ?control the content? or ?influence the meaning or purpose?? Blocking a request or simply failing to answer MIGHT be within the law, but returning a false record certainly seems to me that it would run afoul of the law cited. Owen From theodore at ciscodude.net Thu Sep 15 20:09:03 2016 From: theodore at ciscodude.net (Theodore Baschak) Date: Thu, 15 Sep 2016 15:09:03 -0500 Subject: Any ISPs using AS852 for IP Transit? In-Reply-To: <93F8A4A5-B3A3-47C0-A4AC-C630F1D52278@lixfeld.ca> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <9B17C89B-5FCB-4030-A12A-48FECAF2D090@lixfeld.ca> <20160915191529.GE16464@bamboo.slabnet.com> <93F8A4A5-B3A3-47C0-A4AC-C630F1D52278@lixfeld.ca> Message-ID: I don't think this is standard across the board with Telus. I've also heard (rumours?) of a similar $250 prefix change free associated with Shaw/AS6327 changes before, and also a much larger $750 change prefix change fee with BELL-GT/AS6539, but the customers I know who use them definitely don't get charged these types of fees. Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/ > On Sep 15, 2016, at 2:21 PM, Jason Lixfeld wrote: > > Sure. My question was whether every TELUS BGP customer was being charged for these too, or if I?m the only one. If I?m the only one, then I?m obviously caught in some administrative black hole there that I would like to get myself out of. This is something that has only started happening in the last 6 months or so. Prior to that, we were never charged by them for these requests. Unfortunately, my sales rep has been less than helpful in trying to understand what changed to make us susceptible to these new charges. > >> On Sep 15, 2016, at 3:15 PM, Hugo Slabbert wrote: >> >> So, to be blunt, I would cast this as their charging you NRC for manual work because of their failure to automate this. >> >> -- >> Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com >> pgp key: B178313E | also on Signal >> >> On Thu 2016-Sep-15 15:09:33 -0400, Jason Lixfeld wrote: >> >>> Last time I asked, that wasn?t something that they had implemented, and had no definite plans to do so within any timeframe that was on their radar. >>> >>>> On Sep 15, 2016, at 2:50 PM, Steven Schecter wrote: >>>> >>>> I question their motivation here and would follow up by asking if they support filtering by IRRdb and are merely trying to encourage the practice? >>>> >>>> >>>> /Steve >>>> >>>> On Thu, Sep 15, 2016 at 2:07 PM, Jason Lixfeld wrote: >>>> If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I?d be interested in hearing from you. >>>> >>>> I?d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we?re the only ones? >>>> >>>> Thanks in advance! >>>> >>>> >>>> >>>> -- >>>> Steven J. Schecter >>>> (m) 917.676.1646 >>> > From marka at isc.org Thu Sep 15 21:45:58 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 16 Sep 2016 07:45:58 +1000 Subject: QWEST.NET can you fix your nameservers In-Reply-To: Your message of "Thu, 15 Sep 2016 12:43:57 -0400." References: <20160915073135.2670C54534F2@rock.dv.isc.org> Message-ID: <20160915214558.E2F5C5455C04@rock.dv.isc.org> In message , William Herrin writes: > On Thu, Sep 15, 2016 at 12:22 PM, Aaron C. de Bruyn wrot > e: > > On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews wrote: > >> QWEST isn't the only DNS provider that has broken nameservers. One > >> shouldn't have to try and contact every DNS operator to get them to > >> use protocol compliant servers. > > > > Save yourself some time. Contact the DNS software vendors. ;) > > I'd bet he already has. This looks like a name-and-shame to me, and > probably deserved. > > -Bill Aaron, How am I supposed to know which DNS vendor to contact? DNS server fingerprinting is not a exact science. After that I then still need to work out how to contact every operator of a broken server and get them to contact the DNS vendor to get a fix. And by the way the SOA RNAME is often a blackhole or it bounces or it is syntactically invalid. The best way to get this fixed would be for nameservers to be checked for protocol compliance, by the parent zone operators or their proxies regularly. That the child zone operator be given a short (< 3 months) to fix it then all zones with that server get removed from the parent zone until the server is fixed (apply the final step in the complaints proceedures from RFC 1033) which forces the owner of the zone to fix the server or to move to someone who follows the protocol. The servers for new delegations be checked immediately and the delegation not proceed unless the delegated servers are protocol compliant. Everybody seems to think they know how to write a DNS server. The problem is that most people don't test anything other than simple queries and that includes many of the DNS vendors. Think about all the load balancer vendors that don't handle anything but a A query or only handle A and AAAA queries don't handle DNSKEY queries. There really is no excuse to not handle non-meta qtypes properly (no error not data or name error depending upon whether the name exists or not). My bet is the DNS vendor has issued a update already and that it hasn't been applied. If not Qwest can inform them that their product is broken. Fixing this should be about 10 minutes for the DNS vendor then QA. If you (collectively) haven't already checked your servers go to https://ednscomp.isc.org and check your servers. While you are there look at some of the reports. If there are any tech reporters out there can you report on the issue of non compliance in DNS servers and that it can lead to lookups failing. This issue affects everybody. Mark > -- > William Herrin ................ herrin at dirtside.com bill at herrin.us > Owner, Dirtside Systems ......... Web: -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From aaron at heyaaron.com Thu Sep 15 22:07:50 2016 From: aaron at heyaaron.com (Aaron C. de Bruyn) Date: Thu, 15 Sep 2016 15:07:50 -0700 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <20160915214558.E2F5C5455C04@rock.dv.isc.org> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> Message-ID: On Thu, Sep 15, 2016 at 2:45 PM, Mark Andrews wrote: > > Aaron, > How am I supposed to know which DNS vendor to contact? DNS > Sorry--I should have added a /sarcasm tag. :) > The best way to get this fixed would be for nameservers to be checked > for protocol compliance, by the parent zone operators or their > proxies regularly. That the child zone operator be given a short > (< 3 months) to fix it then all zones with that server get removed > from the parent zone until the server is fixed (apply the final > step in the complaints proceedures from RFC 1033) which forces the > owner of the zone to fix the server or to move to someone who follows > the protocol. The servers for new delegations be checked immediately > and the delegation not proceed unless the delegated servers are > protocol compliant. > Seems a bit harsh, but I'm new to the conversation. What is being out of compliance actually hurting other than the nameserver operator and the zones they host? > My bet is the DNS vendor has issued a update already and that it > hasn't been applied. If not Qwest can inform them that their product > is broken. Fixing this should be about 10 minutes for the DNS > vendor then QA. > Yeah, but the business upgrade cycles are the killer. Why dedicate resources to fix it unless there's a pretty clear line-of-sight to lost profits? That's why so many of my clients refuse to upgrade away from XP. It still works for what they basically need, and it's not really impacting their profit in a way the CFO can directly see. (i.e. he doesn't see people like me who will walk out of a dental office and never come back when I see a 2-plus-year-out-of-date XP machine handling patient information.) I'm sure the same is happening in a large bureaucracy like Qwest. Maybe you're right with a harsher penalty. Be standards compliant or you'll get a warning, then be cut off. > If you (collectively) haven't already checked your servers go to > https://ednscomp.isc.org and check your servers. While you are > there look at some of the reports. > Tested. I'm compliant. I definitely think more comprehensive tools that are easily accessible to admins and CFOs would help. For example, when I explain various zone-related things to CFOs, I'll use http://intodns.com/. It's sorta flashy, and contains some sorta helpful information that a CFO can sorta understand. And a big red 'X' when someone is wrong. Unfortunately it doesn't do DNSSEC. For that, there's another tool. ...and if you want EDNS testing, there's your tool. A tool that tests compliance for everything and spits out errors, warnings, and recommendations might go a long ways towards getting people to solve the problem. Just my $0.02. Nice graphs by the way. -A From aaron at heyaaron.com Thu Sep 15 22:26:39 2016 From: aaron at heyaaron.com (Aaron C. de Bruyn) Date: Thu, 15 Sep 2016 15:26:39 -0700 Subject: ___Your___$ __l O O O Walmart___GiftCard In-Reply-To: References: <20160915073135.2670C54534F2@rock.dv.isc.org> <13615.147395996455@turing-police.cc.vt.edu> Message-ID: That's interesting. heyaaron.com is one big huge catch-all that funnels into my Google Apps for Domains mailbox. There's one account, it has a good password, and it's protected by a Ubikey. I'd be interested in seeing a copy of the headers from that e-mail. -A On Thu, Sep 15, 2016 at 3:15 PM, Brian majors wrote: > I am reporting you to the Fed scammer > > On Sep 15, 2016 6:10 PM, "__WLM__" wrote: > > > > Congarts____This__is Your____$ l O O O ____GiftCard > From marka at isc.org Thu Sep 15 23:30:56 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 16 Sep 2016 09:30:56 +1000 Subject: QWEST.NET can you fix your nameservers In-Reply-To: Your message of "Thu, 15 Sep 2016 15:07:50 -0700." References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> Message-ID: <20160915233057.3671554564B6@rock.dv.isc.org> In message , "Aaron C. de Bruyn" writes: > > On Thu, Sep 15, 2016 at 2:45 PM, Mark Andrews wrote: > > > > Aaron, > > How am I supposed to know which DNS vendor to contact? DNS > > > > Sorry--I should have added a /sarcasm tag. :) > > > > The best way to get this fixed would be for nameservers to be checked > > for protocol compliance, by the parent zone operators or their > > proxies regularly. That the child zone operator be given a short > > (< 3 months) to fix it then all zones with that server get removed > > from the parent zone until the server is fixed (apply the final > > step in the complaints proceedures from RFC 1033) which forces the > > owner of the zone to fix the server or to move to someone who follows > > the protocol. The servers for new delegations be checked immediately > > and the delegation not proceed unless the delegated servers are > > protocol compliant. > > > > Seems a bit harsh, but I'm new to the conversation. What is being out of > compliance actually hurting other than the nameserver operator and the > zones they host? So your helpdesks don't get problem reports when people can't look up domain names? Recursive DNS vendors don't get bug reports when domain names can't be looked up. We don't get fixes developed because there are too many broken servers out there. Because some servers don't answer EDNS requests this leads to false positives on servers not support EDNS when they do. This in turn leads to DNSSEC validation failures as you don't get DNSSEC answers without EDNS. IPv6 deployment was put back years because AAAA DNS lookups got wrong answers. DANE deployment is slow because DNS servers give bad answers to _._tcp./TLSA. Then there is SPF. A fare portion of the reason why the SPF record failed, despite it being architectually cleaner than using TXT records, is that some nameservers gave bad responses to SPF queries. I could go find more examples of the cost of non DNS protocol compliance. > > My bet is the DNS vendor has issued a update already and that it > > hasn't been applied. If not Qwest can inform them that their product > > is broken. Fixing this should be about 10 minutes for the DNS > > vendor then QA. > > > > Yeah, but the business upgrade cycles are the killer. > Why dedicate resources to fix it unless there's a pretty clear > line-of-sight to lost profits? > That's why so many of my clients refuse to upgrade away from XP. It still > works for what they basically need, and it's not really impacting their > profit in a way the CFO can directly see. (i.e. he doesn't see people like > me who will walk out of a dental office and never come back when I see a > 2-plus-year-out-of-date XP machine handling patient information.) > > I'm sure the same is happening in a large bureaucracy like Qwest. > > Maybe you're right with a harsher penalty. Be standards compliant or > you'll get a warning, then be cut off. > > > > > If you (collectively) haven't already checked your servers go to > > https://ednscomp.isc.org and check your servers. While you are > > there look at some of the reports. > > > > Tested. I'm compliant. I definitely think more comprehensive tools that > are easily accessible to admins and CFOs would help. > > For example, when I explain various zone-related things to CFOs, I'll use > http://intodns.com/. It's sorta flashy, and contains some sorta helpful > information that a CFO can sorta understand. > > And a big red 'X' when someone is wrong. > > Unfortunately it doesn't do DNSSEC. For that, there's another tool. > ...and if you want EDNS testing, there's your tool. > > A tool that tests compliance for everything and spits out errors, warnings, > and recommendations might go a long ways towards getting people to solve > the problem. > > Just my $0.02. > > Nice graphs by the way. > > -A > > --001a11394e2c845079053c9314bd > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > >
On T= > hu, Sep 15, 2016 at 2:45 PM, Mark Andrews < mailto:marka at isc.org" target=3D"_blank">marka at isc.org> wrote:= >
x #ccc solid;padding-left:1ex">Aaron,
> =C2=A0 =C2=A0 =C2=A0 =C2=A0How am I supposed to know which DNS vendor to co= > ntact?=C2=A0 DNS

Sorry--I should have a= > dded a /sarcasm tag. =C2=A0:)
=C2=A0
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le= > ft:1ex">The best way to get this fixed would be for nameservers to be check= > ed
> for protocol compliance, by the parent zone operators or their
> proxies regularly.=C2=A0 That the child zone operator be given a short
> (< 3 months) to fix it then all zones with that server get removed
> from the parent zone until the server is fixed (apply the final
> step in the complaints proceedures from RFC 1033) which forces the
> owner of the zone to fix the server or to move to someone who follows
> the protocol.=C2=A0 The servers for new delegations be checked immediately<= > br> > and the delegation not proceed unless the delegated servers are
> protocol compliant.

Seems a bit harsh, = > but I'm new to the conversation.=C2=A0 What is being out of
c= > ompliance actually hurting other than the nameserver operator and the
= >
zones they host?

=C2=A0
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= > ding-left:1ex">My bet is the DNS vendor has issued a update already and tha= > t it
> hasn't been applied.=C2=A0 If not Qwest can inform them that their prod= > uct
> is broken.=C2=A0 Fixing this should be about 10 minutes for the DNS
> vendor then QA.

Yeah, but the business = > upgrade cycles are the killer.
Why dedicate resources to fix it u= > nless there's a pretty clear line-of-sight to lost profits?
T= > hat's why so many of my clients refuse to upgrade away from XP.=C2=A0 I= > t still works for what they basically need, and it's not really impacti= > ng their profit in a way the CFO can directly see. =C2=A0(i.e. he doesn'= > ;t see people like me who will walk out of a dental office and never come b= > ack when I see a 2-plus-year-out-of-date XP machine handling patient inform= > ation.)

I'm sure the same is happening in a la= > rge bureaucracy like Qwest.

Maybe you're right= > with a harsher penalty.=C2=A0 Be standards compliant or you'll get a w= > arning, then be cut off.

=C2=A0
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;= > padding-left:1ex"> > If you (collectively) haven't already checked your servers go to
> h= > ttps://ednscomp.isc.org and check your servers.=C2=A0 While you are
> there look at some of the reports.

Test= > ed.=C2=A0 I'm compliant.=C2=A0 I definitely think more comprehensive to= > ols that are easily accessible to admins and CFOs would help.
>
For example, when I explain various zone-related things to CFOs= > , I'll use http://intodns.com/.=C2= > =A0 It's sorta flashy, and contains some sorta helpful information that= > a CFO can sorta understand.

And a big red 'X&= > #39; when someone is wrong.

Unfortunately it doesn= > 't do DNSSEC.=C2=A0 For that, there's another tool.
...an= > d if you want EDNS testing, there's your tool.

>A tool that tests compliance for everything and spits out errors, warnings= > , and recommendations might go a long ways towards getting people to solve = > the problem.

Just my $0.02.

iv>Nice graphs by the way.

-A
iv> > > --001a11394e2c845079053c9314bd-- -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From eric-list at truenet.com Thu Sep 15 23:39:45 2016 From: eric-list at truenet.com (Eric Tykwinski) Date: Thu, 15 Sep 2016 19:39:45 -0400 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <20160915233057.3671554564B6@rock.dv.isc.org> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> <20160915233057.3671554564B6@rock.dv.isc.org> Message-ID: <9442FCB1-E039-4EDD-8A0F-F5F351BC99B4@truenet.com> Ironically, I always wondered why I was told not to publish SPF records, since it did make more sense to have both, and slowly remove the TXT records later. Thanks for the heads up? What do you think really is best practice now? Sincerely, Eric Tykwinski TrueNet, Inc. P: 610-429-8300 > On Sep 15, 2016, at 7:30 PM, Mark Andrews wrote: > > So your helpdesks don't get problem reports when people can't look > up domain names? Recursive DNS vendors don't get bug reports when > domain names can't be looked up. We don't get fixes developed > because there are too many broken servers out there. > > Because some servers don't answer EDNS requests this leads to false > positives on servers not support EDNS when they do. This in turn > leads to DNSSEC validation failures as you don't get DNSSEC answers > without EDNS. > > IPv6 deployment was put back years because AAAA DNS lookups got > wrong answers. > > DANE deployment is slow because DNS servers give bad answers to > _._tcp./TLSA. > > Then there is SPF. A fare portion of the reason why the SPF record > failed, despite it being architectually cleaner than using TXT > records, is that some nameservers gave bad responses to SPF queries. > > I could go find more examples of the cost of non DNS protocol > compliance. > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From bill at herrin.us Thu Sep 15 23:59:45 2016 From: bill at herrin.us (William Herrin) Date: Thu, 15 Sep 2016 19:59:45 -0400 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <20160915233057.3671554564B6@rock.dv.isc.org> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> <20160915233057.3671554564B6@rock.dv.isc.org> Message-ID: On Thu, Sep 15, 2016 at 7:30 PM, Mark Andrews wrote: > Then there is SPF. A fare portion of the reason why the SPF record > failed, despite it being architectually cleaner than using TXT > records, is that some nameservers gave bad responses to SPF queries. Hi Mark, I'm going to stop you there. The SPF record type failed because resolvers can't pass requests between clients and authoritative servers unless they can parse them. New DNS record types essentially require a universal software upgrade before they work. And universal software upgrades do not happen well or in anything approaching a timely manner. The next new DNS record type will fail for the same reason. And the one after that. TXT is the DNS record type that's extensible without a software upgrade. Like it or lump it. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From marka at isc.org Fri Sep 16 00:04:05 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 16 Sep 2016 10:04:05 +1000 Subject: QWEST.NET can you fix your nameservers In-Reply-To: Your message of "Thu, 15 Sep 2016 19:39:45 -0400." <9442FCB1-E039-4EDD-8A0F-F5F351BC99B4@truenet.com> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> <20160915233057.3671554564B6@rock.dv.isc.org> <9442FCB1-E039-4EDD-8A0F-F5F351BC99B4@truenet.com> Message-ID: <20160916000405.5E1D254567EE@rock.dv.isc.org> In message <9442FCB1-E039-4EDD-8A0F-F5F351BC99B4 at truenet.com>, Eric Tykwinski w rites: > Ironically, I always wondered why I was told not to publish SPF records, > since it did make more sense to have both, and slowly remove the TXT > records later. Thanks for the heads up??? > > What do you think really is best practice now? For SPF the decision was made to stay with TXT. The IAB wrote RFC 5507 - Design Choices When Expanding the DNS. As for testing there is: https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue-04 There is general consensus that the tests are correct to the level that they cover. There can always be more tests added. There is less consensus on how we get from where we are now to where we need to be. The EDNS tests tool was the starting point for this draft. Mark > Sincerely, > > Eric Tykwinski > TrueNet, Inc. > P: 610-429-8300 > > > On Sep 15, 2016, at 7:30 PM, Mark Andrews wrote: > > > > So your helpdesks don't get problem reports when people can't look > > up domain names? Recursive DNS vendors don't get bug reports when > > domain names can't be looked up. We don't get fixes developed > > because there are too many broken servers out there. > > > > Because some servers don't answer EDNS requests this leads to false > > positives on servers not support EDNS when they do. This in turn > > leads to DNSSEC validation failures as you don't get DNSSEC answers > > without EDNS. > > > > IPv6 deployment was put back years because AAAA DNS lookups got > > wrong answers. > > > > DANE deployment is slow because DNS servers give bad answers to > > _._tcp./TLSA. > > > > Then there is SPF. A fare portion of the reason why the SPF record > > failed, despite it being architectually cleaner than using TXT > > records, is that some nameservers gave bad responses to SPF queries. > > > > I could go find more examples of the cost of non DNS protocol > > compliance. > > -- > > Mark Andrews, ISC > > 1 Seymour St., Dundas Valley, NSW 2117, Australia > > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From jfmezei_nanog at vaxination.ca Fri Sep 16 00:07:49 2016 From: jfmezei_nanog at vaxination.ca (Jean-Francois Mezei) Date: Thu, 15 Sep 2016 20:07:49 -0400 Subject: Lawsuits for falsyfying DNS responses ? In-Reply-To: <7B8AC8E8-83C2-4BD9-AA34-4212DF995822@delong.com> References: <57D6E8BC.9010404@vaxination.ca> <57D8CE99.1080101@vaxination.ca> <7B8AC8E8-83C2-4BD9-AA34-4212DF995822@delong.com> Message-ID: <57DB37D5.2000008@vaxination.ca> On 2016-09-15 16:03, Owen DeLong wrote: > Please explain to me how one modifies a request or response without > managing to ?control the content? or ?influence the meaning or purpose?? > > Blocking a request or simply failing to answer MIGHT be within the law, > but returning a false record certainly seems to me that it would run afoul > of the law cited. Blocking would also be a form of control. Because Section 36 has a "unless authorized by CRTC" escape clause, one has to show to the CRTC that granting permission would be bad. Since court proceedings have already begun, it is likely the CRTC will be involved in court, at which point, the more evidence they have, the more chances they have of arguing against the QC loterry censorship. From marka at isc.org Fri Sep 16 00:28:42 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 16 Sep 2016 10:28:42 +1000 Subject: QWEST.NET can you fix your nameservers In-Reply-To: Your message of "Thu, 15 Sep 2016 19:59:45 -0400." References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> <20160915233057.3671554564B6@rock.dv.isc.org> Message-ID: <20160916002842.5B65554569B3@rock.dv.isc.org> In message , William Herrin writes: > On Thu, Sep 15, 2016 at 7:30 PM, Mark Andrews wrote: > > Then there is SPF. A fare portion of the reason why the SPF record > > failed, despite it being architectually cleaner than using TXT > > records, is that some nameservers gave bad responses to SPF queries. > > Hi Mark, > > I'm going to stop you there. The SPF record type failed because > resolvers can't pass requests between clients and authoritative > servers unless they can parse them. New DNS record types essentially > require a universal software upgrade before they work. And universal > software upgrades do not happen well or in anything approaching a > timely manner. The next new DNS record type will fail for the same > reason. And the one after that. Again lack of DNS compliance. Go read STD 13 then tell me that Microsoft ships a standards compliant resolver. They still don't last time I checked. Libresolv could look up any tuple from back when the UCB developed it. You *never* needed universal support for new types. It is a myth. You just need the authoritative servers to support the type and the client to support the type. Everything else treated it as a opaque blob. That is why compression pointers were only allowed for well known types. That is why records have a length field. What STD 13 missed was a presentation format for unknown types. There were implementations that got that wrong, including named, but we fixed that well before SPF ever became a issue. We have RFC 3597 which allows authoritative servers to load and save records they don't know the internal structure of. This was published in September 2003. All the major DNS vendors support it. This addressed the oversight in STD 13. You have lazy operators that haven't designed their web tools to support RFC 3597. Mark > TXT is the DNS record type that's extensible without a software > upgrade. Like it or lump it. > > Regards, > Bill Herrin > > > -- > William Herrin ................ herrin at dirtside.com bill at herrin.us > Owner, Dirtside Systems ......... Web: -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From dot at dotat.at Fri Sep 16 11:32:36 2016 From: dot at dotat.at (Tony Finch) Date: Fri, 16 Sep 2016 12:32:36 +0100 Subject: QWEST.NET can you fix your nameservers In-Reply-To: <20160915214558.E2F5C5455C04@rock.dv.isc.org> References: <20160915073135.2670C54534F2@rock.dv.isc.org> <20160915214558.E2F5C5455C04@rock.dv.isc.org> Message-ID: Mark Andrews wrote: > > My bet is the DNS vendor has issued a update already and that it > hasn't been applied. $ fpdns sauthns1.qwest.net. fingerprint (sauthns1.qwest.net., 63.150.72.5): NLnetLabs NSD 3.1.0 -- 3.2.8 [New Rules] fingerprint (sauthns1.qwest.net., 2001:428:0:0:0:0:0:7): NLnetLabs NSD 3.1.0 -- 3.2.8 [New Rules] $ dig +nocookie +noall +answer version.bind ch txt @sauthns1.qwest.net. version.bind. 0 CH TXT "3.2.2" https://www.nlnetlabs.nl/projects/nsd/ NSD 3.2.2 - May 18, 2009 Tony. -- f.anthony.n.finch http://dotat.at/ - I xn--zr8h punycode Humber, Thames, Dover, Wight: Cyclonic at first, but mainly northerly 5 to 7, decreasing 4 at times later. Slight or moderate. Occasional rain, perhaps thundery at first, fog patches at first in Humber. Moderate or poor, occasionally very poor in Humber. From simon at slimey.org Fri Sep 16 13:12:46 2016 From: simon at slimey.org (Simon Lockhart) Date: Fri, 16 Sep 2016 14:12:46 +0100 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: <20160916131246.GP29651@dilbert.slimey.org> All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon From nanog at ics-il.net Fri Sep 16 13:28:21 2016 From: nanog at ics-il.net (Mike Hammett) Date: Fri, 16 Sep 2016 08:28:21 -0500 (CDT) Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <1367618559.8005.1474032497813.JavaMail.mhammett@ThunderFuck> A network that doesn't support IPv6, yet discriminates against CGNAT? That seems like a promising future. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Simon Lockhart" To: nanog at nanog.org Sent: Friday, September 16, 2016 8:12:46 AM Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon From rdobbins at arbor.net Fri Sep 16 13:32:12 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Fri, 16 Sep 2016 20:32:12 +0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> On 16 Sep 2016, at 20:12, Simon Lockhart wrote: > Has anyone else come up against the problem, and/or have any > suggestions on how best to resolve it? I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. ----------------------------------- Roland Dobbins From simon at slimey.org Fri Sep 16 13:38:00 2016 From: simon at slimey.org (Simon Lockhart) Date: Fri, 16 Sep 2016 14:38:00 +0100 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> References: <20160916131246.GP29651@dilbert.slimey.org> <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> Message-ID: <20160916133800.GQ29651@dilbert.slimey.org> On Fri Sep 16, 2016 at 08:32:12PM +0700, Roland Dobbins wrote: > Another aspect is ensuring that one has the ability to detect, classify, > traceback, and mitigate outbound badness southbound of the CGN. Unless PSN can tell us what traffic they consider bad, how can we detect and classify it? We certainly have the ability to traceback and mitigate, once we know what we're looking for. My understanding of the issue is that there are infected PCs on our network, which are being used as part of a distributed attack, but at the application layer, rather than network layer - distributed password brute-force, or similar. Unless we know what to look for, it's hard to detect and stop it. Simon From rdobbins at arbor.net Fri Sep 16 13:49:23 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Fri, 16 Sep 2016 20:49:23 +0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916133800.GQ29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> <20160916133800.GQ29651@dilbert.slimey.org> Message-ID: <8B399DC0-3655-4452-A96B-E934844FE602@arbor.net> On 16 Sep 2016, at 20:38, Simon Lockhart wrote: > Unless we know what to look for, it's hard to detect and stop it. It's not just application-layer stuff - they're subject to all sorts of attacks. Screening out the obvious stuff would certainly help. The main issue is a dearth of engagement of clueful folks in the global operational community. Some gaming-oriented networks are well-represented; others are not, sadly. ----------------------------------- Roland Dobbins From A.L.M.Buxey at lboro.ac.uk Fri Sep 16 13:49:17 2016 From: A.L.M.Buxey at lboro.ac.uk (A.L.M.Buxey at lboro.ac.uk) Date: Fri, 16 Sep 2016 13:49:17 +0000 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <20160916134917.GG22492@lboro.ac.uk> Hi, as others have said, need to engage with one of their other units to get this sorted out - as a network provider, their customers are relying on YOU to access their service, PSN should care. technically, you could start looking at netflows to the PSN and see if anyone is engaged in DDoS via that route...and , if you offer IPv6 native service to end users, ask PSN when they are going to be offer an IPv6 service to their users - so this CGNAT stuff can go ;-) alan From dougm.work at gmail.com Fri Sep 16 15:31:59 2016 From: dougm.work at gmail.com (Doug Montgomery) Date: Fri, 16 Sep 2016 11:31:59 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <2AB19C93-0E35-4AA0-B781-EA4653A729A8@beckman.org> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> <2AB19C93-0E35-4AA0-B781-EA4653A729A8@beckman.org> Message-ID: Ah, the global system I was referring to was the RPKI as distributed repository of routing information. With consistent properties (data formats, security models, data validation techniques, etc) across all 5 RIRs. What an ISP does with the RPKI data, interns of route filtering, is always a local policy decision. Somehow "global enforcement system" sounded like a vision where ISPs don't have a choice of how and where to use the system. Maybe I read too much into the phrasing. In the end, I think the issues boils down to if the community has the desire and will to address the route hijack issue. If the answer is no, then no further discussion is needed. If the answer is yes, then it is best to discuss and compare real proposals / mechanisms to address the issue at scale. I am still interested in some detail on the "tyranny implications". We can't address them until we know what they are and how they compare to any other solution that tries to address the same problem. dougm On Wed, Sep 14, 2016 at 6:04 PM, Mel Beckman wrote: > Doug, > > I was basing my comments on your statement "If only there were a global > system.." However you slice or dice it, the tyranny implications have not > yet been addressed. That certainly needs to be in front of any technical > idea such as RPKI. > > Although I haven't participated in the OT&E, nothing I've read in RFC 6810 > talks about these issues. It talks about authentication and transport > security, but doesn't talk about the potential for government interference. > > -mel beckman > > On Sep 14, 2016, at 8:22 AM, Doug Montgomery wrote: > > Mel, > > If you are speaking of RPKI based origin validation, I am not sure > "automated / global enforcement system" is a useful description. It does > provide a consistent means for address holders to declare AS's authorized > to announce prefixes, and a means for remote ASs to compare received > updates vs such declarations. What the receiving AS does with the > validation information is strictly a local policy matter. > > Frankly, this is no more a "new automated enforcement system" than > IRR-based route filtering has been for 20 years. The only difference is > that there is a consistent security model across all 5 RIRs as to who can > make such declarations and it is tightly tied to the address allocation > business process. > > I have seen a lot of FUD about the specter of interference, but not a lot > of serious thought / discussion. Having a serious technical discussion of > potential risks and mitigations in the system would be useful. > > dougm > > On Wed, Sep 14, 2016 at 10:51 AM, Mel Beckman wrote: > >> Scott and Doug, >> >> The problem with a new automated enforcement system is that it hobbles >> both agility and innovation. ISPs have enjoyed simple BGP management, >> entirely self-regulated, for decades. A global enforcement system, besides >> being dang hard to do correctly, brings the specter of government >> interference, since such a system could be overtaken by government entities >> to manhandle free speech. >> >> In my opinion, the community hasn't spent nearly enough time discussing >> the danger aspect. Being engineers, we focus on technical means, ignoring >> the fact that we're designing our own guillotine. >> >> -mel beckman >> >> > On Sep 14, 2016, at 12:10 AM, Scott Weeks >> wrote: >> > >> > >> > >> > --- dougm.work at gmail.com wrote: >> > From: Doug Montgomery >> > >> > If only there were a global system, with consistent and verifiable >> security >> > properties, to permit address holders to declare the set of AS's >> authorized >> > to announce their prefixes, and routers anywhere on the Internet to >> > independently verify the corresponding validity of received >> announcements. >> > >> > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* >> > ------------------------------------------------ >> > >> > >> > Yes, RPKI. That's what I was waiting for. Now we can get to >> > a real discussion... ;-) >> > >> > scott >> > > > > -- > DougM at Work > > -- DougM at Work From cb.list6 at gmail.com Fri Sep 16 16:02:48 2016 From: cb.list6 at gmail.com (Ca By) Date: Fri, 16 Sep 2016 09:02:48 -0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: On Friday, September 16, 2016, Simon Lockhart wrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon Here is a picture of what you are experiencing http://test-ipv6.com/faq_avoids_ipv6.html Sometimes people need pictures to understand why IPv6 is important From mel at beckman.org Fri Sep 16 16:06:19 2016 From: mel at beckman.org (Mel Beckman) Date: Fri, 16 Sep 2016 16:06:19 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> <2AB19C93-0E35-4AA0-B781-EA4653A729A8@beckman.org>, Message-ID: <488F4F3C-277E-4F34-8F31-920254FABEDE@beckman.org> Doug, Although RPKI is voluntary and decisions are local, those decisions are also automated. DNS is voluntary, and decisions are local as well, yet the government has been able to leverage DNS to unilaterally seize domain names without due process. Like Maxwell's Demons, it's theoretically possible for ISPs everywhere to notice government malfeasance and rush to a unified decision to counter it. But in practice this never happens. Preventing government manhandling needs to be a design goal. -mel beckman On Sep 16, 2016, at 8:32 AM, Doug Montgomery > wrote: Ah, the global system I was referring to was the RPKI as distributed repository of routing information. With consistent properties (data formats, security models, data validation techniques, etc) across all 5 RIRs. What an ISP does with the RPKI data, interns of route filtering, is always a local policy decision. Somehow "global enforcement system" sounded like a vision where ISPs don't have a choice of how and where to use the system. Maybe I read too much into the phrasing. In the end, I think the issues boils down to if the community has the desire and will to address the route hijack issue. If the answer is no, then no further discussion is needed. If the answer is yes, then it is best to discuss and compare real proposals / mechanisms to address the issue at scale. I am still interested in some detail on the "tyranny implications". We can't address them until we know what they are and how they compare to any other solution that tries to address the same problem. dougm On Wed, Sep 14, 2016 at 6:04 PM, Mel Beckman > wrote: Doug, I was basing my comments on your statement "If only there were a global system.." However you slice or dice it, the tyranny implications have not yet been addressed. That certainly needs to be in front of any technical idea such as RPKI. Although I haven't participated in the OT&E, nothing I've read in RFC 6810 talks about these issues. It talks about authentication and transport security, but doesn't talk about the potential for government interference. -mel beckman On Sep 14, 2016, at 8:22 AM, Doug Montgomery > wrote: Mel, If you are speaking of RPKI based origin validation, I am not sure "automated / global enforcement system" is a useful description. It does provide a consistent means for address holders to declare AS's authorized to announce prefixes, and a means for remote ASs to compare received updates vs such declarations. What the receiving AS does with the validation information is strictly a local policy matter. Frankly, this is no more a "new automated enforcement system" than IRR-based route filtering has been for 20 years. The only difference is that there is a consistent security model across all 5 RIRs as to who can make such declarations and it is tightly tied to the address allocation business process. I have seen a lot of FUD about the specter of interference, but not a lot of serious thought / discussion. Having a serious technical discussion of potential risks and mitigations in the system would be useful. dougm On Wed, Sep 14, 2016 at 10:51 AM, Mel Beckman > wrote: Scott and Doug, The problem with a new automated enforcement system is that it hobbles both agility and innovation. ISPs have enjoyed simple BGP management, entirely self-regulated, for decades. A global enforcement system, besides being dang hard to do correctly, brings the specter of government interference, since such a system could be overtaken by government entities to manhandle free speech. In my opinion, the community hasn't spent nearly enough time discussing the danger aspect. Being engineers, we focus on technical means, ignoring the fact that we're designing our own guillotine. -mel beckman > On Sep 14, 2016, at 12:10 AM, Scott Weeks > wrote: > > > > --- dougm.work at gmail.com wrote: > From: Doug Montgomery > > > If only there were a global system, with consistent and verifiable security > properties, to permit address holders to declare the set of AS's authorized > to announce their prefixes, and routers anywhere on the Internet to > independently verify the corresponding validity of received announcements. > > *cough https://www.nanog.org/meetings/abstract?id=2846 cough* > ------------------------------------------------ > > > Yes, RPKI. That's what I was waiting for. Now we can get to > a real discussion... ;-) > > scott -- DougM at Work -- DougM at Work From cscora at apnic.net Fri Sep 16 18:01:26 2016 From: cscora at apnic.net (Routing Analysis Role Account) Date: Sat, 17 Sep 2016 04:01:26 +1000 (AEST) Subject: Weekly Routing Table Report Message-ID: <20160916180126.690AE7D9D@thyme.apnic.net> This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats at lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith . Routing Table Report 04:00 +10GMT Sat 17 Sep, 2016 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 609829 Prefixes after maximum aggregation (per Origin AS): 220069 Deaggregation factor: 2.77 Unique aggregates announced (without unneeded subnets): 297950 Total ASes present in the Internet Routing Table: 54857 Prefixes per ASN: 11.12 Origin-only ASes present in the Internet Routing Table: 36396 Origin ASes announcing only one prefix: 15402 Transit ASes present in the Internet Routing Table: 6516 Transit-only ASes present in the Internet Routing Table: 168 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 54 Max AS path prepend of ASN ( 55644) 51 Prefixes from unregistered ASNs in the Routing Table: 60 Unregistered ASNs in the Routing Table: 15 Number of 32-bit ASNs allocated by the RIRs: 15444 Number of 32-bit ASNs visible in the Routing Table: 11945 Prefixes from 32-bit ASNs in the Routing Table: 47636 Number of bogon 32-bit ASNs visible in the Routing Table: 84 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 347 Number of addresses announced to Internet: 2826656228 Equivalent to 168 /8s, 123 /16s and 89 /24s Percentage of available address space announced: 76.3 Percentage of allocated address space announced: 76.3 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.3 Total number of prefixes smaller than registry allocations: 197571 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 156454 Total APNIC prefixes after maximum aggregation: 42942 APNIC Deaggregation factor: 3.64 Prefixes being announced from the APNIC address blocks: 169988 Unique aggregates announced from the APNIC address blocks: 68960 APNIC Region origin ASes present in the Internet Routing Table: 5194 APNIC Prefixes per ASN: 32.73 APNIC Region origin ASes announcing only one prefix: 1156 APNIC Region transit ASes present in the Internet Routing Table: 946 Average APNIC Region AS path length visible: 4.4 Max APNIC Region AS path length visible: 54 Number of APNIC region 32-bit ASNs visible in the Routing Table: 2380 Number of APNIC addresses announced to Internet: 759640004 Equivalent to 45 /8s, 71 /16s and 47 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 183986 Total ARIN prefixes after maximum aggregation: 89627 ARIN Deaggregation factor: 2.05 Prefixes being announced from the ARIN address blocks: 189745 Unique aggregates announced from the ARIN address blocks: 88444 ARIN Region origin ASes present in the Internet Routing Table: 16204 ARIN Prefixes per ASN: 11.71 ARIN Region origin ASes announcing only one prefix: 5699 ARIN Region transit ASes present in the Internet Routing Table: 1705 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 1467 Number of ARIN addresses announced to Internet: 1106648544 Equivalent to 65 /8s, 246 /16s and 29 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 145716 Total RIPE prefixes after maximum aggregation: 71812 RIPE Deaggregation factor: 2.03 Prefixes being announced from the RIPE address blocks: 156009 Unique aggregates announced from the RIPE address blocks: 96512 RIPE Region origin ASes present in the Internet Routing Table: 18132 RIPE Prefixes per ASN: 8.60 RIPE Region origin ASes announcing only one prefix: 7832 RIPE Region transit ASes present in the Internet Routing Table: 3032 Average RIPE Region AS path length visible: 4.4 Max RIPE Region AS path length visible: 27 Number of RIPE region 32-bit ASNs visible in the Routing Table: 5048 Number of RIPE addresses announced to Internet: 708290048 Equivalent to 42 /8s, 55 /16s and 166 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 61300 Total LACNIC prefixes after maximum aggregation: 12400 LACNIC Deaggregation factor: 4.94 Prefixes being announced from the LACNIC address blocks: 77048 Unique aggregates announced from the LACNIC address blocks: 37423 LACNIC Region origin ASes present in the Internet Routing Table: 2473 LACNIC Prefixes per ASN: 31.16 LACNIC Region origin ASes announcing only one prefix: 541 LACNIC Region transit ASes present in the Internet Routing Table: 583 Average LACNIC Region AS path length visible: 4.8 Max LACNIC Region AS path length visible: 22 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 2789 Number of LACNIC addresses announced to Internet: 170641728 Equivalent to 10 /8s, 43 /16s and 201 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-266652 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 14676 Total AfriNIC prefixes after maximum aggregation: 3278 AfriNIC Deaggregation factor: 4.48 Prefixes being announced from the AfriNIC address blocks: 16692 Unique aggregates announced from the AfriNIC address blocks: 6293 AfriNIC Region origin ASes present in the Internet Routing Table: 734 AfriNIC Prefixes per ASN: 22.74 AfriNIC Region origin ASes announcing only one prefix: 174 AfriNIC Region transit ASes present in the Internet Routing Table: 181 Average AfriNIC Region AS path length visible: 4.6 Max AfriNIC Region AS path length visible: 18 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 261 Number of AfriNIC addresses announced to Internet: 81097728 Equivalent to 4 /8s, 213 /16s and 116 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5553 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3514 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 3113 11142 1062 KIXS-AS-KR Korea Telecom, KR 17974 2945 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2688 1496 528 BSNL-NIB National Internet Backbone, IN 9808 2250 8781 42 CMNET-GD Guangdong Mobile Communication 4755 2052 429 218 TATACOMM-AS TATA Communications formerl 4808 1761 2295 528 CHINA169-BJ China Unicom Beijing Provin 45899 1727 1235 89 VNPT-AS-VN VNPT Corp, VN 24560 1542 510 221 AIRTELBROADBAND-AS-AP Bharti Airtel Ltd Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 22773 3514 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 6389 2219 3671 41 BELLSOUTH-NET-BLK - BellSouth.net Inc., 18566 2195 405 110 MEGAPATH5-US - MegaPath Corporation, US 20115 1938 1977 404 CHARTER-NET-HKY-NC - Charter Communicat 30036 1738 337 332 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 209 1717 5070 652 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 6983 1676 848 227 ITCDELTA - Earthlink, Inc., US 16509 1394 2532 466 AMAZON-02 - Amazon.com, Inc., US 7018 1370 20092 1008 ATT-INTERNET4 - AT&T Services, Inc., US 11492 1288 230 622 CABLEONE - CABLE ONE, INC., US Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3329 169 15 ALJAWWALSTC-AS , SA 20940 2791 1059 1980 AKAMAI-ASN1 , US 34984 1974 327 356 TELLCOM-AS , TR 12479 1331 1018 46 UNI2-AS , ES 13188 1234 99 62 BANKINFORM-AS , UA 8551 1206 377 46 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 6849 1144 355 21 UKRTELNET , UA 8402 1078 544 15 CORBINA-AS Russia, RU 12389 925 1113 401 ROSTELECOM-AS , RU 6830 886 2752 465 LGI-UPC formerly known as UPC Broadband Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 10620 3479 539 195 Telmex Colombia S.A., CO 8151 2275 3382 561 Uninet S.A. de C.V., MX 7303 1544 955 245 Telecom Argentina S.A., AR 6503 1397 437 55 Axtel, S.A.B. de C.V., MX 11830 1327 367 57 Instituto Costarricense de Electricidad 7738 994 1882 40 Telemar Norte Leste S.A., BR 3816 966 467 206 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 28573 911 2181 167 CLARO S.A., BR 11172 907 125 76 Alestra, S. de R.L. de C.V., MX 18881 883 1036 22 TELEF?NICA BRASIL S.A, BR Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1179 402 50 LINKdotNET-AS, EG 36903 684 344 115 MT-MPLS, MA 37611 668 51 5 Afrihost, ZA 36992 547 1373 25 ETISALAT-MISR, EG 8452 543 1472 15 TE-AS TE-AS, EG 37492 384 250 69 ORANGE-TN, TN 24835 345 610 16 RAYA-AS, EG 29571 301 37 12 CITelecom-AS, CI 15399 297 35 6 WANANCHI-KE, KE 2018 269 327 74 TENET-1, ZA Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5553 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3514 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 22773 3514 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3479 539 195 Telmex Colombia S.A., CO 39891 3329 169 15 ALJAWWALSTC-AS , SA 4766 3113 11142 1062 KIXS-AS-KR Korea Telecom, KR 17974 2945 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 20940 2791 1059 1980 AKAMAI-ASN1 , US 9829 2688 1496 528 BSNL-NIB National Internet Backbone, IN 8151 2275 3382 561 Uninet S.A. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 22773 3514 3369 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 39891 3329 3314 ALJAWWALSTC-AS , SA 10620 3479 3284 Telmex Colombia S.A., CO 7545 3514 3264 TPG-INTERNET-AP TPG Telecom Limited, AU 17974 2945 2867 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9808 2250 2208 CMNET-GD Guangdong Mobile Communication 6389 2219 2178 BELLSOUTH-NET-BLK - BellSouth.net Inc., 9829 2688 2160 BSNL-NIB National Internet Backbone, IN 18566 2195 2085 MEGAPATH5-US - MegaPath Corporation, US 4766 3113 2051 KIXS-AS-KR Korea Telecom, KR Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65001 PRIVATE 5.143.176.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.200.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.208.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.216.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65000 PRIVATE 31.219.177.0/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65000 PRIVATE 31.219.177.128/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65512 PRIVATE 45.252.244.0/24 45899 VNPT-AS-VN VNPT Corp, VN 65512 PRIVATE 45.252.245.0/24 45899 VNPT-AS-VN VNPT Corp, VN Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 23.249.144.0/20 40430 COLO4JAX-AS - colo4jax, LLC, US 24.49.144.0/20 30164 USACOMMUNICATIONS-1 - USA COMPANIES, LP 27.100.7.0/24 56096 UNKNOWN 41.73.1.0/24 37004 -Reserved AS-, ZZ 41.73.2.0/24 37004 -Reserved AS-, ZZ 41.73.3.0/24 37004 -Reserved AS-, ZZ 41.73.4.0/24 37004 -Reserved AS-, ZZ 41.73.5.0/24 37004 -Reserved AS-, ZZ 41.73.6.0/24 37004 -Reserved AS-, ZZ 41.73.7.0/24 37004 -Reserved AS-, ZZ Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:16 /9:13 /10:36 /11:101 /12:269 /13:518 /14:1052 /15:1768 /16:13136 /17:7840 /18:12986 /19:25332 /20:38742 /21:40307 /22:67944 /23:59536 /24:338713 /25:531 /26:511 /27:344 /28:55 /29:32 /30:13 /31:1 /32:33 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 39891 2896 3329 ALJAWWALSTC-AS , SA 22773 2739 3514 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2087 2195 MEGAPATH5-US - MegaPath Corporation, US 30036 1552 1738 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 6389 1418 2219 BELLSOUTH-NET-BLK - BellSouth.net Inc., 10620 1401 3479 Telmex Colombia S.A., CO 6983 1326 1676 ITCDELTA - Earthlink, Inc., US 34984 1257 1974 TELLCOM-AS , TR 11492 1190 1288 CABLEONE - CABLE ONE, INC., US 13188 1011 1234 BANKINFORM-AS , UA Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1631 2:776 4:22 5:2169 6:27 8:990 12:1803 13:42 14:1740 15:47 16:2 17:95 18:126 20:50 22:1 23:1836 24:1804 27:2325 31:1781 32:85 33:4 35:5 36:323 37:2311 38:1254 39:36 40:97 41:2909 42:452 43:1859 44:47 45:2140 46:2554 47:95 49:1212 50:905 51:13 52:547 54:344 55:6 56:7 57:42 58:1569 59:1015 60:376 61:1821 62:1514 63:1923 64:4547 65:2179 66:4266 67:2205 68:1118 69:3256 70:1282 71:527 72:2031 74:2547 75:399 76:413 77:1380 78:1210 79:908 80:1293 81:1400 82:988 83:708 84:856 85:1565 86:486 87:1125 88:545 89:2086 90:204 91:6061 92:947 93:2365 94:2448 95:2472 96:521 97:340 98:939 99:90 100:78 101:1152 103:12319 104:2469 105:108 106:432 107:1403 108:671 109:2257 110:1269 111:1625 112:1054 113:1271 114:1078 115:1683 116:1659 117:1553 118:2031 119:1584 120:936 121:1118 122:2266 123:2070 124:1571 125:1835 128:670 129:438 130:414 131:1353 132:611 133:175 134:485 135:192 136:387 137:409 138:1895 139:424 140:574 141:456 142:662 143:933 144:735 145:165 146:942 147:653 148:1391 149:531 150:665 151:884 152:666 153:297 154:676 155:946 156:526 157:524 158:391 159:1232 160:511 161:733 162:2394 163:544 164:773 165:1118 166:334 167:1183 168:2175 169:669 170:1920 171:254 172:701 173:1686 174:758 175:669 176:1726 177:3996 178:2234 179:1186 180:2157 181:1767 182:2051 183:996 184:827 185:7399 186:3202 187:2142 188:2166 189:1767 190:7346 191:1272 192:9037 193:5724 194:4456 195:3855 196:1771 197:1146 198:5567 199:5760 200:7116 201:3597 202:10124 203:9822 204:4512 205:2744 206:2972 207:3076 208:4044 209:3862 210:3887 211:2036 212:2739 213:2332 214:870 215:70 216:5791 217:1979 218:805 219:609 220:1631 221:885 222:700 223:1305 End of report From tony at wicks.co.nz Fri Sep 16 21:40:04 2016 From: tony at wicks.co.nz (Tony Wicks) Date: Sat, 17 Sep 2016 09:40:04 +1200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <003001d21062$e3f2b820$abd82860$@wicks.co.nz> So the pain has finally flowed down to other parts of the world. (APNIC ran out of IP's a long time ago, so CGN has been in use here for a lot longer) This issue is one I have been dealing with for the last four years. Only with Sony, no other company has caused such a headache in regard to CGNAT. I will not go into the long and painful saga of dealing with the constant issue of Sony putting blocks on random pool addresses, refusing to supply sufficient information to identify rouge users (timestamp, source IP, destination IP and port) then telling our customers it is a problem at the ISP end, but... Something happened about three months ago that Proves that if the Sony technical people want to get off their asses they are perfectly capable of supplying adequate information to identify a rogue user for the ISP to deal with. One of the local Sony PSN helpline managers actually managed to convince one of their technical people to supply a spreadsheet that magically contained sufficient information to allow us to identify a couple of users that did indeed have multiple infections. Great I thought, now if we can just get them to automate/regularly sent this info we will have a way forward. Alas, it appears it was a one off and we are back to the start. I will quote below what the Sony Network guy said when explaining why they can't send detailed information every time - " From: SNEI-NOC-Abuse [mailto:SNEI-NOC-Abuse at am.sony.com] Sent: Thursday, 11 August 2016 8:38 AM To: ##me## Cc: ##helpful Sony guy## Subject: RE: PSN / Flip Network blocks Hello, There is quite a bit of extra computing power required to produce the CSV file with timestamps and destination IP addresses. We send out over 6000 emails per day which already takes a significant amount of resources and time. We tend to get around 20-30 responses. Instead of wasting the resources on all those emails we generate CSV files for those who respond. We hope you understand. Thank you for taking action on these." So there you go, Sony can indeed solve this issue, but apparently a company that makes computers has insufficient computing power and staff to do so. Oh and after this, despite being asked many times they have never responded to requests for the CSV or similar detailed info. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Simon Lockhart Sent: Saturday, 17 September 2016 1:13 AM To: nanog at nanog.org Subject: PlayStationNetwork blocking of CGNAT public addresses All, We operate an access network with several hundred thousand users. Increasingly we're putting the users behind CGNAT in order to continue to give them an IPv4 service (we're all dual-stack, so they all get public IPv6 too). Due to the demographic of our users, many of them are gamers. We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our CGNAT outside addresses, because they claim to have received anomalous, or 'attack' traffic from that IP. This obviously causes problems for the other legitimate users who end up behind the same public IPv4 address. Despite numerous attempts to engage with PSN, they are unwilling to give us any additional information which would allow us to identify the 'rogue' users on our network, or to identify the 'unwanted' traffic so that we could either block it, or use it to identify the rogue users ourselves. Has anyone else come up against the problem, and/or have any suggestions on how best to resolve it? Many thanks in advance, Simon From patrick.sumby at sohonet.com Fri Sep 16 01:55:06 2016 From: patrick.sumby at sohonet.com (Patrick Sumby) Date: Thu, 15 Sep 2016 18:55:06 -0700 Subject: Anyone with a clue at Zayo? Message-ID: Have a turnup we've been working on all day and no luck so far. Now we're being told that nobody can help outside hours :( Any help much appreciated. Thanks Pat From michalis.bersimis at hq.cyta.gr Fri Sep 16 13:41:22 2016 From: michalis.bersimis at hq.cyta.gr (michalis.bersimis at hq.cyta.gr) Date: Fri, 16 Sep 2016 13:41:22 +0000 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> References: <20160916131246.GP29651@dilbert.slimey.org> <63D9F23B-F136-46F0-8A3F-05AA099E1C98@arbor.net> Message-ID: <0c842eec8a7b440fb41929347d5f99a3@PLAMAIL01.corp.cyta> Another aspect, for those users that need to go the PSN network but experience issues via the CGNAT, an opt-out solution (giving them public IPv4) may should mitigate the problem, that PSN network does not support IPv6. After all what percentage of your total subscribers that uses PSN and are gamers 2-3% ? Which might be relatively small amount to give public IPv4. Michalis -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Roland Dobbins Sent: Friday, September 16, 2016 4:32 PM To: nanog at nanog.org Subject: Re: PlayStationNetwork blocking of CGNAT public addresses On 16 Sep 2016, at 20:12, Simon Lockhart wrote: > Has anyone else come up against the problem, and/or have any > suggestions on how best to resolve it? I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate. Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN. This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse. AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks. ----------------------------------- Roland Dobbins From JKrejci at usinternet.com Sat Sep 17 00:35:48 2016 From: JKrejci at usinternet.com (Justin Krejci) Date: Sat, 17 Sep 2016 00:35:48 +0000 Subject: Anyone with a clue at Zayo? In-Reply-To: References: Message-ID: <3E9C67DA261AC349B60FF3609F5E211D7D75B375@USI-2K10EX01-MT.usicorp.usinternet.com> Might help if you indicate type of service as they have lots of services covered by different groups: IP transit, wave, dark fiber, voip, Colo, etc. Their Enterprise division does yet other services. Might also help if you provide at least a general location/region. -----Original Message----- From: Patrick Sumby [patrick.sumby at sohonet.com] Received: Friday, 16 Sep 2016, 7:22PM To: nanog at nanog.org [nanog at nanog.org] Subject: Anyone with a clue at Zayo? Have a turnup we've been working on all day and no luck so far. Now we're being told that nobody can help outside hours :( Any help much appreciated. Thanks Pat From mohta at necom830.hpcl.titech.ac.jp Sat Sep 17 00:54:46 2016 From: mohta at necom830.hpcl.titech.ac.jp (Masataka Ohta) Date: Sat, 17 Sep 2016 09:54:46 +0900 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: Simon Lockhart wrote: > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? The best solution is to have a common practice on a set of public port numbers assigned to a host behind NAT. For example, with a practice that, if a port in a range between N*8 and N*8+7 is assigned to a host, other ports in the range is not assigned to other hosts, service providers can block packets based on IP addresses and ranges, especially if correspondence between hosts and ranges are rather stable. But, it may be too late to make such practice common, I'm afraid. Or, wait for a while until service providers receive enough amount of feedback from innocent users. To accelerate it, you can make correspondence between hosts and public addresses not so stable, which makes almost all your IP addresses marked bad quickly, which may make you loss some customer, unless other ISPs also do so. Masataka Ohta From ops.lists at gmail.com Sat Sep 17 12:39:31 2016 From: ops.lists at gmail.com (Suresh Ramasubramanian) Date: Sat, 17 Sep 2016 18:09:31 +0530 Subject: One more thing to watch out for at data centers - fire drills Message-ID: <619CEDC6-48DD-4654-A140-266EACD714AE@gmail.com> http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours?utm_source=bbcfb Releasing inert gas from fire suppression units that were over pressurized resulted in an extremely loud noise ? causing cabinets full of hard drives to vibrate ? which got transmitted to the read ? write heads of the drives. Amazing sort of outage + data loss, and this time the physical security plant chief gets to write up the RCA. --srs From sp at iphh.net Sat Sep 17 13:16:33 2016 From: sp at iphh.net (Sascha Pollok) Date: Sat, 17 Sep 2016 15:16:33 +0200 Subject: One more thing to watch out for at data centers - fire drills In-Reply-To: <619CEDC6-48DD-4654-A140-266EACD714AE@gmail.com> References: <619CEDC6-48DD-4654-A140-266EACD714AE@gmail.com> Message-ID: <157384a8f68.27cd.0891eb2d685d091c0d1b837b0ec4f146@iphh.net> > http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours?utm_source=bbcfb > > Releasing inert gas from fire suppression units that were over pressurized > resulted in an extremely loud noise ? causing cabinets full of hard drives > to vibrate ? which got transmitted to the read ? write heads of the drives. > > Amazing sort of outage + data loss, and this time the physical security > plant chief gets to write up the RCA. A quick comment: for the N2 systems we built in our datacenters we added mufflers to reduce the noise which raised the price for the system by only 5%. Totally worth it. Yet many datacenters do not add them. -Sascha From hcb at netcases.net Sat Sep 17 13:27:26 2016 From: hcb at netcases.net (hcb at netcases.net) Date: Sat, 17 Sep 2016 09:27:26 -0400 Subject: One more thing to watch out for at data centers - fire drills Message-ID: <39c58385215a7272e5812a447db21bf7@netcases.net> On 2016-09-17 08:39, Suresh Ramasubramanian wrote: > http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours?utm_source=bbcfb > > Releasing inert gas from fire suppression units that were over > pressurized resulted in an extremely loud noise ? causing cabinets > full of hard drives to vibrate ? which got transmitted to the read ? > write heads of the drives. > > Amazing sort of outage + data loss, and this time the physical > security plant chief gets to write up the RCA. > > --srs Another unexpected result when we had an all-out Halon test: thick fog, apparently from cold gas and somewhat humid air. I'm glad to have been watching through windows. Visibility in the room dropped to zero. From math at sizone.org Sat Sep 17 18:12:20 2016 From: math at sizone.org (Ken Chase) Date: Sat, 17 Sep 2016 14:12:20 -0400 Subject: One more thing to watch out for at data centers - fire drills In-Reply-To: <39c58385215a7272e5812a447db21bf7@netcases.net> References: <39c58385215a7272e5812a447db21bf7@netcases.net> Message-ID: <20160917181220.GV30299@sizone.org> All of these discussions sounds infinitely safe for humans. Servers and network gear is replaceable. Sounds like the failure was not one of DC mismanagement but human safety errors. /kc On Sat, Sep 17, 2016 at 09:27:26AM -0400, hcb at netcases.net said: >On 2016-09-17 08:39, Suresh Ramasubramanian wrote: >>http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours?utm_source=bbcfb >> >>Releasing inert gas from fire suppression units that were over >>pressurized resulted in an extremely loud noise ??? causing cabinets >>full of hard drives to vibrate ??? which got transmitted to the read ??? >>write heads of the drives. >> >>Amazing sort of outage + data loss, and this time the physical >>security plant chief gets to write up the RCA. >> >>--srs > >Another unexpected result when we had an all-out Halon test: thick fog, >apparently from cold gas and somewhat humid air. I'm glad to have been >watching through windows. Visibility in the room dropped to zero. -- Ken Chase - math at sizone.org Guelph Canada From larrysheldon at cox.net Sat Sep 17 21:43:18 2016 From: larrysheldon at cox.net (Larry Sheldon) Date: Sat, 17 Sep 2016 16:43:18 -0500 Subject: One more thing to watch out for at data centers - fire drills In-Reply-To: References: Message-ID: On 9/17/2016 07:39, Suresh Ramasubramanian wrote: > http://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours?utm_source=bbcfb > > Releasing inert gas from fire suppression units that were over > pressurized resulted in an extremely loud noise My experience is only with in-specification systems (and only in tape libraries) but those tests were pretty loud. ? causing cabinets > full of hard drives to vibrate ? which got transmitted to the read ? > write heads of the drives. My experiences were back in the days of washing-machine class disc drives and they were a 4-hour fire-wall away, but I don't remember them being impacted. (I can't believe that I was allowed to conduct a test with them running, but I don't remember shutting them down.) I wonder if orientation mattered--mine were all platters parallel to the floor, I wonder if the damaged ones were parallel to the wave front. > full of hard drives to vibrate ? which got transmitted to the read ? > write heads of the drives. > > Amazing sort of outage + data loss, and this time the physical > security plant chief gets to write up the RCA. -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account. From tom.smyth at wirelessconnect.eu Sun Sep 18 12:30:52 2016 From: tom.smyth at wirelessconnect.eu (Tom Smyth) Date: Sun, 18 Sep 2016 13:30:52 +0100 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: Hi Simon, as other responders have said it is an inherent issue with NAT in general, on workaround is to limit the ratio of actual users to an external IPv4 address, the other thing we have seen from our Abuse contact emails from PSN, is that malicious activity towards the PSN is often accompanied by other malicious activities such as SSH brute force outbound and spaming... I would suggest that 1) limit the ratio of users to an external ipv4 address as much as possible (which would reduce the impact of one compromised customer bringing down play time for other clients behind the same nat 2)do some "canary in the mine" monitoring for obviously malicious traffic (loads of SMTP traffic outbound) and lots of connection requests to SSH servers ... if you see that traffic from behind your CGNAT device .. just temporarily block the internal ip of the user until they clean up their devices. this is the pain with NAT you have to do extra work in order prevent infected users interrupting internet connectivity for other innocent users... I think you can use simple firewall rules on your edge router to identify multiple connections to SMTP and SSH in a short period of time.. If you do the minimum to detect that abuse then you cant be accused of invading peoples privacy... (bear in mind obvious false positives) (Monitoring systems etc) ... Hope this helps, On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart wrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon > > -- Kindest regards, Tom Smyth Mobile: +353 87 6193172 --------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL This email contains information which may be confidential or privileged. The information is intended solely for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify me by telephone or by electronic mail immediately. Any opinions expressed are those of the author, not the company's .This email does not constitute either offer or acceptance of any contractually binding agreement. Such offer or acceptance must be communicated in writing. You are requested to carry out your own virus check before opening any attachment. Thomas Smyth accepts no liability for any loss or damage which may be caused by malicious software or attachments. From rsk at gsp.org Sun Sep 18 13:07:03 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Sun, 18 Sep 2016 09:07:03 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <20160918130703.GA17415@gsp.org> On Sun, Sep 18, 2016 at 01:30:52PM +0100, Tom Smyth wrote: > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. Seconded. This is something I've recommended for years (decades, I suppose by now). Simple measurements of what's "normal" for your operation in terms of connection rates, types, etc., are easy to make. That in turn enables measurements of what's abnormal and that in turn enables manual or automatic actions. For example: if the average number of outbound SSH connections established per hour per host across all hosts behind CGNAT is 3.2, and you see a host making 1100/hour: that's a problem. It might be someone who botched a Perl script; or it might be a botted host trying to brute-force its way into something. These kinds of measurements are relatively easy to make and don't require invading user privacy. They won't catch everything, of course, but they're not intended to. They may catch enough to solve the problem in front of you at the moment *and*, if they do that, they may reduce the scope/scale of the rest of the problems to make them more tractable via other techniques. ---rsk From beecher at beecher.cc Sun Sep 18 13:15:08 2016 From: beecher at beecher.cc (Tom Beecher) Date: Sun, 18 Sep 2016 09:15:08 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth wrote: > Hi Simon, > > as other responders have said it is an inherent issue with NAT in general, > on workaround is to limit the ratio of actual users to an external IPv4 > address, the other thing we have seen from our Abuse contact emails from > PSN, is that malicious activity towards the PSN is often accompanied by > other malicious activities such as SSH brute force outbound and spaming... > > I would suggest that > > 1) limit the ratio of users to an external ipv4 address as much as possible > (which would reduce the impact of one compromised customer bringing down > play time for other clients behind the same nat > > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. > > this is the pain with NAT you have to do extra work in order prevent > infected users interrupting internet connectivity for other innocent > users... > I think you can use simple firewall rules on your edge router to identify > multiple connections to SMTP and SSH in a short period of time.. > > If you do the minimum to detect that abuse then you cant be accused of > invading peoples privacy... (bear in mind obvious false positives) > (Monitoring systems etc) ... > > Hope this helps, > > On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart wrote: > > > All, > > > > We operate an access network with several hundred thousand users. > > Increasingly > > we're putting the users behind CGNAT in order to continue to give them an > > IPv4 > > service (we're all dual-stack, so they all get public IPv6 too). Due to > the > > demographic of our users, many of them are gamers. > > > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > > of our > > CGNAT outside addresses, because they claim to have received anomalous, > or > > 'attack' traffic from that IP. This obviously causes problems for the > other > > legitimate users who end up behind the same public IPv4 address. > > > > Despite numerous attempts to engage with PSN, they are unwilling to give > us > > any additional information which would allow us to identify the 'rogue' > > users > > on our network, or to identify the 'unwanted' traffic so that we could > > either > > block it, or use it to identify the rogue users ourselves. > > > > Has anyone else come up against the problem, and/or have any suggestions > on > > how best to resolve it? > > > > Many thanks in advance, > > > > Simon > > > > > > > -- > Kindest regards, > Tom Smyth > > Mobile: +353 87 6193172 > --------------------------------- > PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL > This email contains information which may be confidential or privileged. > The information is intended solely for the use of the individual or entity > named above. If you are not the intended recipient, be aware that > any disclosure, copying, distribution or use of the contents of this > information is prohibited. If you have received this electronic > transmission in error, please notify me by telephone or by electronic mail > immediately. Any opinions expressed are those of the author, not the > company's .This email does not constitute either offer or acceptance of > any contractually binding agreement. Such offer or acceptance must be > communicated in > writing. You are requested to carry out your own virus check before opening > any attachment. Thomas Smyth accepts no liability for any loss or damage > which may be caused by malicious software or attachments. > From nanog at ics-il.net Sun Sep 18 13:19:15 2016 From: nanog at ics-il.net (Mike Hammett) Date: Sun, 18 Sep 2016 08:19:15 -0500 (CDT) Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <631701672.9977.1474204754937.JavaMail.mhammett@ThunderFuck> People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Tom Beecher" To: "Tom Smyth" Cc: "NANOG" Sent: Sunday, September 18, 2016 8:15:08 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses This is, as many things are, a huge problem in communication. Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. ISP says 'Oh crap, sorry, what's going on? We'll run it down.' Sony says nothing. Let's just stop here for a second. This is fundamentally no different then the 'I have a problem, it's the network! complaints we've all dealt with forever. You spend days/weeks/months working on it. Maybe you ultimately find a goofy switchport, or maybe you discover that the server HDDs were crapping the bed and the problem server was chugging because of that. But you had to spend tons of time working on it because you couldn't get the info you need because the reporter was CONVINCED they KNEW what it was. Why should Simon have to spend hours of engineering time fishing through traffic captures and logs when he doesn't even know what he's LOOKING for? What does PSN consider 'abuse' here? Does Simon have customers infected with botnets that are targeting PSN at times? Or does PSN assume nobody will ever have more than a couple Playstations in a house, so if they see more than N connections to PSN from the same IP, it's malicious, since CGN is likely not something they considered? ( If anyone wants to place beer wagers, I'm picking the later. ) I spend about 8 weeks this year going back and forth with a Very Large Website Network who had blocked a /17 of IP space from accessing ANY of their sites because of 'malicious traffic' from a specific /23. 5 of those weeks, their responses consisted of 'it's malicious, you go find it, should be obvious', 'you clearly don't know what you're doing, we're wasting our time', etc. Week 5, I was able to extract that it was a specific web crawler that they said was knocking their databases over. After a conversation with their CIO the following week, they came back and admitted that a junior system admin made some PHP changes on a bunch of servers that he didn't think was in production,and when we crawled THOSE servers, Bad Things Happened for them. We were doing nothing wrong ; they just refused to look, and found it easier to blame us. Simon's getting screwed because he's not being given any information to try and solve the problem, and because his customers are likely blaming him because he's their ISP. Sony needs to stand up and work with him here. On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth wrote: > Hi Simon, > > as other responders have said it is an inherent issue with NAT in general, > on workaround is to limit the ratio of actual users to an external IPv4 > address, the other thing we have seen from our Abuse contact emails from > PSN, is that malicious activity towards the PSN is often accompanied by > other malicious activities such as SSH brute force outbound and spaming... > > I would suggest that > > 1) limit the ratio of users to an external ipv4 address as much as possible > (which would reduce the impact of one compromised customer bringing down > play time for other clients behind the same nat > > 2)do some "canary in the mine" monitoring for obviously malicious traffic > (loads of SMTP traffic outbound) and lots of connection requests to SSH > servers ... if you see that traffic from behind your CGNAT device .. just > temporarily block the internal ip of the user until they clean up their > devices. > > this is the pain with NAT you have to do extra work in order prevent > infected users interrupting internet connectivity for other innocent > users... > I think you can use simple firewall rules on your edge router to identify > multiple connections to SMTP and SSH in a short period of time.. > > If you do the minimum to detect that abuse then you cant be accused of > invading peoples privacy... (bear in mind obvious false positives) > (Monitoring systems etc) ... > > Hope this helps, > > On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart wrote: > > > All, > > > > We operate an access network with several hundred thousand users. > > Increasingly > > we're putting the users behind CGNAT in order to continue to give them an > > IPv4 > > service (we're all dual-stack, so they all get public IPv6 too). Due to > the > > demographic of our users, many of them are gamers. > > > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > > of our > > CGNAT outside addresses, because they claim to have received anomalous, > or > > 'attack' traffic from that IP. This obviously causes problems for the > other > > legitimate users who end up behind the same public IPv4 address. > > > > Despite numerous attempts to engage with PSN, they are unwilling to give > us > > any additional information which would allow us to identify the 'rogue' > > users > > on our network, or to identify the 'unwanted' traffic so that we could > > either > > block it, or use it to identify the rogue users ourselves. > > > > Has anyone else come up against the problem, and/or have any suggestions > on > > how best to resolve it? > > > > Many thanks in advance, > > > > Simon > > > > > > > -- > Kindest regards, > Tom Smyth > > Mobile: +353 87 6193172 > --------------------------------- > PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL > This email contains information which may be confidential or privileged. > The information is intended solely for the use of the individual or entity > named above. If you are not the intended recipient, be aware that > any disclosure, copying, distribution or use of the contents of this > information is prohibited. If you have received this electronic > transmission in error, please notify me by telephone or by electronic mail > immediately. Any opinions expressed are those of the author, not the > company's .This email does not constitute either offer or acceptance of > any contractually binding agreement. Such offer or acceptance must be > communicated in > writing. You are requested to carry out your own virus check before opening > any attachment. Thomas Smyth accepts no liability for any loss or damage > which may be caused by malicious software or attachments. > From list at satchell.net Sun Sep 18 13:45:51 2016 From: list at satchell.net (Stephen Satchell) Date: Sun, 18 Sep 2016 06:45:51 -0700 Subject: One more thing to watch out for at data centers - fire drills In-Reply-To: References: Message-ID: On 09/17/2016 02:43 PM, Larry Sheldon wrote: > My experiences were back in the days of washing-machine class disc > drives and they were a 4-hour fire-wall away, but I don't remember them > being impacted. (I can't believe that I was allowed to conduct a test > with them running, but I don't remember shutting them down.) > > I wonder if orientation mattered--mine were all platters parallel to the > floor, I wonder if the damaged ones were parallel to the wave front. If you watched the video of the guy who screamed at his disk drives to cause temporary faults, the JBOD had its platters horizontal to the floor. One of the reason the washing-machine-sized CDC Storage Module Drives weren't affected by high noise level is the sheer beefy mass of the head assembly and the voice coil. Also, the track spacing on the platters of those drives was far less dense, so any noise-induced mis-tracking would be minuscule, and easily handled by said voice-coil's position-error system. The heads were larger, as well as the head arms. In this situation, mass is your friend. From fw at deneb.enyo.de Sun Sep 18 13:56:30 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 18 Sep 2016 15:56:30 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160918130703.GA17415@gsp.org> (Rich Kulawiec's message of "Sun, 18 Sep 2016 09:07:03 -0400") References: <20160916131246.GP29651@dilbert.slimey.org> <20160918130703.GA17415@gsp.org> Message-ID: <87k2e91fch.fsf@mid.deneb.enyo.de> * Rich Kulawiec: > For example: if the average number of outbound SSH connections > established per hour per host across all hosts behind CGNAT is 3.2, > and you see a host making 1100/hour: that's a problem. It might be > someone who botched a Perl script; or it might be a botted host > trying to brute-force its way into something. If you do this, you break Github. (If I guess Simon's network correctly, then I've seen reports which suggest that they might already be doing this.) From fw at deneb.enyo.de Sun Sep 18 13:58:57 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 18 Sep 2016 15:58:57 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: (Tom Beecher's message of "Sun, 18 Sep 2016 09:15:08 -0400") References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <87fuox1f8e.fsf@mid.deneb.enyo.de> * Tom Beecher: > Simon's getting screwed because he's not being given any information to try > and solve the problem, and because his customers are likely blaming him > because he's their ISP. We don't know that for sure. Another potential issue is that the ISP just cannot afford to notify its compromised customers, even if they were able to detect them. From simon at slimey.org Sun Sep 18 14:06:50 2016 From: simon at slimey.org (Simon Lockhart) Date: Sun, 18 Sep 2016 15:06:50 +0100 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <87fuox1f8e.fsf@mid.deneb.enyo.de> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> Message-ID: <20160918140650.GR29651@dilbert.slimey.org> On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote: > * Tom Beecher: > > Simon's getting screwed because he's not being given any information to try > > and solve the problem, and because his customers are likely blaming him > > because he's their ISP. > > We don't know that for sure. Another potential issue is that the ISP > just cannot afford to notify its compromised customers, even if they > were able to detect them. I'd like to think that we're pretty responsive to taking our users offline when they're compromised and we're made aware of it - either through our own tools, or through 3rd party notifications. The process with Sony goes something like: - User reports they can't reach PSN - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked us" - We say "Okay, that's a CGNAT public IP, can you help us identify the which inside user that is - (timestamp,ip,port) logs, or some way to identify the bad traffic so we can look for it ourselves" - Sony say no, either through silence, or explicitly. - We have unhappy user(s), who blame us. Simon From beecher at beecher.cc Sun Sep 18 15:07:05 2016 From: beecher at beecher.cc (Tom Beecher) Date: Sun, 18 Sep 2016 11:07:05 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <87fuox1f8e.fsf@mid.deneb.enyo.de> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> Message-ID: An email to a user notifying them they're likely compromised costs basically nothing. An email to their entire subscriber base also costs nothing. If you find me an ISP that can't afford to notify users, I'll show you one that shouldn't be in business anyways. There's this presumption of guilt here, that Sony is right, and Simon's subscribers are doing something malicious, yet they won't provide any evidence of that. Even if they didn't know what it was, come back with 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during these times. We're not quite sure what it is, but it's causing [problem X].' It would still be a question of maliciousness or not, but it would be something to work with. Providing nothing just perpetuates this finger pointing game, and nothing gets solved. On Sun, Sep 18, 2016 at 9:58 AM, Florian Weimer wrote: > * Tom Beecher: > > > Simon's getting screwed because he's not being given any information to > try > > and solve the problem, and because his customers are likely blaming him > > because he's their ISP. > > We don't know that for sure. Another potential issue is that the ISP > just cannot afford to notify its compromised customers, even if they > were able to detect them. > From fw at deneb.enyo.de Sun Sep 18 15:13:32 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 18 Sep 2016 17:13:32 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: (Tom Beecher's message of "Sun, 18 Sep 2016 11:07:05 -0400") References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> Message-ID: <871t0hz1er.fsf@mid.deneb.enyo.de> * Tom Beecher: > An email to a user notifying them they're likely compromised costs > basically nothing. If this increases the probability that the customer contacts customer support, in some markets, there is a risk that the account will never turn profitable during the current contract period. (Granted, my information may be woefully out of date, but my impression is that price-based competition is still pretty much cut-throat over here.) > If you find me an ISP that can't afford to notify users, I'll show > you one that shouldn't be in business anyways. I'm not blaming the ISP. (I may have done so in the past.) If we end up in such a situation, it's hardly the fault of one single ISP. > There's this presumption of guilt here, that Sony is right, and Simon's > subscribers are doing something malicious, yet they won't provide any > evidence of that. Even if they didn't know what it was, come back with > 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during > these times. We're not quite sure what it is, but it's causing [problem > X].' It would still be a question of maliciousness or not, but it would be > something to work with. Providing nothing just perpetuates this finger > pointing game, and nothing gets solved. Yes, indeed. Resolving most networking problems needs cooperation, because at the most basic level, the Internet is still about connecting otherwise unrelated networks. From fw at deneb.enyo.de Sun Sep 18 15:17:33 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 18 Sep 2016 17:17:33 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160918140650.GR29651@dilbert.slimey.org> (Simon Lockhart's message of "Sun, 18 Sep 2016 15:06:50 +0100") References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> Message-ID: <87wpi9xmnm.fsf@mid.deneb.enyo.de> * Simon Lockhart: > On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote: >> * Tom Beecher: >> > Simon's getting screwed because he's not being given any information to try >> > and solve the problem, and because his customers are likely blaming him >> > because he's their ISP. >> >> We don't know that for sure. Another potential issue is that the ISP >> just cannot afford to notify its compromised customers, even if they >> were able to detect them. > > I'd like to think that we're pretty responsive to taking our users offline > when they're compromised and we're made aware of it - either through our own > tools, or through 3rd party notifications. Okay, then perhaps my guess of the ISP involved is wrong. > The process with Sony goes something like: > > - User reports they can't reach PSN > - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked > us" > - We say "Okay, that's a CGNAT public IP, can you help us identify the which > inside user that is - (timestamp,ip,port) logs, or some way to identify the > bad traffic so we can look for it ourselves" > - Sony say no, either through silence, or explicitly. > - We have unhappy user(s), who blame us. Yes, that's not very constructive. Out of curiosity, how common is end-to-end reporting of source/destination port information (in addition to source IP addresses and destination IP addresses)? Have the anti-abuse mechanisms finalyl caught on with CGNAT, or is it possible that the PSN operator themselves do not have such detailed data? From simon at slimey.org Sun Sep 18 15:26:43 2016 From: simon at slimey.org (Simon Lockhart) Date: Sun, 18 Sep 2016 16:26:43 +0100 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <87wpi9xmnm.fsf@mid.deneb.enyo.de> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> <87wpi9xmnm.fsf@mid.deneb.enyo.de> Message-ID: <20160918152642.GS29651@dilbert.slimey.org> On Sun Sep 18, 2016 at 05:17:33PM +0200, Florian Weimer wrote: > Okay, then perhaps my guess of the ISP involved is wrong. It's not hard to find out who I work for :) > Out of curiosity, how common is end-to-end reporting of > source/destination port information (in addition to source IP > addresses and destination IP addresses)? Have the anti-abuse > mechanisms finalyl caught on with CGNAT, or is it possible that the > PSN operator themselves do not have such detailed data? 99.99% of abuse reports we receive contain the information, but that's because 99.99% of abuse reports we receive are from the 'copyright police', and their tools capture and include it in the reports. Once you discard that 99.99%, and are left with the stuff that is worthy of manual investigation, I'd say that almost all of it only contains timestamp and source IP. Sometimes it'll also contain destination IP (so we can take a best guess based on netflow data), and very occasionally it'll also contain source port information. I'd say the same also applies to requests for information that we receive from law enforcement agencies. In most cases, they're working from weblogs, and I'd be tempted to say that most webservers' 'out of the box' configuration does not log source port, only source IP in the web access logs. Simon From larrysheldon at cox.net Sun Sep 18 21:26:21 2016 From: larrysheldon at cox.net (Larry Sheldon) Date: Sun, 18 Sep 2016 16:26:21 -0500 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <5c5b1ceb-c77c-12c1-f4b4-7ec81eb0bc79@cox.net> On 9/18/2016 08:19, Mike Hammett wrote: > People love to hate incumbent telcos because of their arrogance (and > frankly it's deserved), but people forget that big content can be > just as arrogant and just as deserving of hatred. I never did see the benefit or the approach. To anybody. -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account. From larrysheldon at cox.net Sun Sep 18 21:48:43 2016 From: larrysheldon at cox.net (Larry Sheldon) Date: Sun, 18 Sep 2016 16:48:43 -0500 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <98c13320-5a46-b624-6add-e3ba56d5ecf1@cox.net> On 9/18/2016 16:26, Larry Sheldon wrote: > > > On 9/18/2016 08:19, Mike Hammett wrote: >> People love to hate incumbent telcos because of their arrogance (and >> frankly it's deserved), but people forget that big content can be >> just as arrogant and just as deserving of hatred. > > I never did see the benefit or the approach. To anybody. > I never did see the benefit oF the approach. To anybody. -- "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid." --Albert Einstein From Larry's Cox account. From tony at wicks.co.nz Sun Sep 18 22:41:59 2016 From: tony at wicks.co.nz (Tony Wicks) Date: Mon, 19 Sep 2016 10:41:59 +1200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160918140650.GR29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> Message-ID: <009301d211fd$e07f6890$a17e39b0$@wicks.co.nz> Interestingly, Sony (SNEI-NOC-Abuse - Sony say no, either through silence, or explicitly. From Valdis.Kletnieks at vt.edu Sun Sep 18 22:58:07 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Sun, 18 Sep 2016 18:58:07 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <009301d211fd$e07f6890$a17e39b0$@wicks.co.nz> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> <009301d211fd$e07f6890$a17e39b0$@wicks.co.nz> Message-ID: <71969.1474239487@turing-police.cc.vt.edu> On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said: > Interestingly, Sony (SNEI-NOC-Abuse replied to being forwarded back one of their notification blocks requesting > more detailed information with a csv file in under an hour! So I guess name-and-shame *does* work? :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From jason at thebaughers.com Mon Sep 19 00:09:16 2016 From: jason at thebaughers.com (Jason Baugher) Date: Sun, 18 Sep 2016 19:09:16 -0500 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <71969.1474239487@turing-police.cc.vt.edu> References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> <009301d211fd$e07f6890$a17e39b0$@wicks.co.nz> <71969.1474239487@turing-police.cc.vt.edu> Message-ID: So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request. It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about. On Sun, Sep 18, 2016 at 5:58 PM, wrote: > On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said: > > Interestingly, Sony (SNEI-NOC-Abuse jut > > replied to being forwarded back one of their notification blocks > requesting > > more detailed information with a csv file in under an hour! > > So I guess name-and-shame *does* work? :) > From tony at wicks.co.nz Mon Sep 19 00:28:33 2016 From: tony at wicks.co.nz (Tony Wicks) Date: Mon, 19 Sep 2016 12:28:33 +1200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> <87fuox1f8e.fsf@mid.deneb.enyo.de> <20160918140650.GR29651@dilbert.slimey.org> <009301d211fd$e07f6890$a17e39b0$@wicks.co.nz> <71969.1474239487@turing-police.cc.vt.edu> Message-ID: <009c01d2120c$c385d080$4a917180$@wicks.co.nz> So the last one we successfully managed to isolate, our customer they had more than one PC with multiple infections. It?s not Playstation?s, but Windows machines that are infected with I assume some malware that is trying to log into PSN. cheers From: Jason Baugher [mailto:jason at thebaughers.com] Sent: Monday, 19 September 2016 12:09 PM To: Valdis.Kletnieks at vt.edu Cc: Tony Wicks ; NANOG Subject: Re: PlayStationNetwork blocking of CGNAT public addresses So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request. It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about. On Sun, Sep 18, 2016 at 5:58 PM, > wrote: On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said: > Interestingly, Sony (SNEI-NOC-Abuse sony dot com) jut > replied to being forwarded back one of their notification blocks requesting > more detailed information with a csv file in under an hour! So I guess name-and-shame *does* work? :) From morrowc.lists at gmail.com Mon Sep 19 01:13:37 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Sun, 18 Sep 2016 21:13:37 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <488F4F3C-277E-4F34-8F31-920254FABEDE@beckman.org> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <54AAA3D5-80EC-4154-9378-1BD7ED8F474F@beckman.org> <2AB19C93-0E35-4AA0-B781-EA4653A729A8@beckman.org> <488F4F3C-277E-4F34-8F31-920254FABEDE@beckman.org> Message-ID: On Fri, Sep 16, 2016 at 12:06 PM, Mel Beckman wrote: > > Preventing government manhandling needs to be a design goal. > Can you proffer some potential solutions or directions to look? At the end of the day the ISP or DNS operator or Enterprise is subject to local law enforcement action(s), so I'm not sure there's a remedy to your design goal. From beecher at beecher.cc Mon Sep 19 02:25:44 2016 From: beecher at beecher.cc (Tom Beecher) Date: Sun, 18 Sep 2016 22:25:44 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> Message-ID: So after reading your explanation of things... Your technical protections for your client proved sufficient to handle the attack. You took OFFENSIVE action by hijacking the IP space. By your own statements, it was only in response to threats against your company. You were no longer providing DDoS protection to a client. You were exacting a vendetta against someone who was being MEAN to you. Even if that person probably deserved it, you still cannot do what was done. I appreciate the desire to want to protect friends and family from anonymous threats, and also realize how ill equipped law enforcement usually is while something like this is occurring. However, in my view, by taking the action you did, you have shown your company isn't ready to be operating in the security space. Being threatened by bad actors is a nominal part of doing business in the security space. Unfortunately you didn't handle it well, and I think that will stick to you for a long time. On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend wrote: > @ca & Matt - No, we do not plan to ever intentionally perform a > non-authorized BGP hijack in the future. > > @Steve - Correct, the attack had already been mitigated. The decision to > hijack the attackers IP space was to deal with their threats, which if > carried through could have potentially lead to physical harm. Although the > hijack gave us a unique insight into the attackers services, it was not a > factor that influenced my decision. > > @Blake & Mel - We will likely cover some of these questions in a future > blog post. > From onetrueseanrose at gmail.com Sat Sep 17 21:32:57 2016 From: onetrueseanrose at gmail.com (Sean Rose) Date: Sat, 17 Sep 2016 23:32:57 +0200 Subject: "Defensive" BGP hijacking? Message-ID: I know Bryant Townsend (ex staminus employee), Marshal Webb (aka m_nerva, lulzsec informant) and others from backconnect.net performed a similar BGP hijacking against staminus earlier this year. https://bgpstream.com/event/21051 Shortly afterwards, on 10th of march a zine is released leaking the Staminus user database and contents of several customer servers. The times aren't the only interesting factor here, even the format of the release just screams m_nerva. Zines are very rare these days. So rare in fact that the last similar zine before the staminus hack was released in 2013 by HTP, a hacker group m_nerva was loosely affiliated with during it's early days. I *strongly* believe Bryant Townsend and Marshal Webb hacked Staminus and produced the "Fuck 'em all." zine Sean Rose From onetrueseanrose at gmail.com Sat Sep 17 22:16:26 2016 From: onetrueseanrose at gmail.com (Sean Rose) Date: Sun, 18 Sep 2016 00:16:26 +0200 Subject: "Defensive" BGP hijacking? In-Reply-To: References: Message-ID: And here's the final bit. I'd like to think that is 100% conclusive proof of what happened. The IP range hijacked by backconnect.net, 72.20.0.0/24 returns interesting results on google: https://staminus.thecthulhu.com/zine.txt ## Global allows ALLOW_MAIN="" ALLOW_MAIN="$ALLOW_MAIN $RFC1918 $LOCAL" ALLOW_MAIN="$ALLOW_MAIN 72.20.1.2 72.20.0.0/24 69.197.1.0/24" # Internal Backconnect.net hijacked Staminus's internal management range 72.20.0.0/24 and used that to gain further access to Staminus's systems. On Sat, Sep 17, 2016 at 11:32 PM, Sean Rose wrote: > I know Bryant Townsend (ex staminus employee), Marshal Webb (aka m_nerva, > lulzsec informant) and others from backconnect.net performed a similar > BGP hijacking against staminus earlier this year. > > https://bgpstream.com/event/21051 > > Shortly afterwards, on 10th of march a zine is released leaking the > Staminus user database and contents of several customer servers. > > The times aren't the only interesting factor here, even the format of the > release just screams m_nerva. Zines are very rare these days. So rare in > fact that the last similar zine before the staminus hack was released in > 2013 by HTP, a hacker group m_nerva was loosely affiliated with during it's > early days. > > I *strongly* believe Bryant Townsend and Marshal Webb hacked Staminus and > produced the "Fuck 'em all." zine > > > Sean Rose > From ml-nanog at techcompute.net Mon Sep 19 03:59:43 2016 From: ml-nanog at techcompute.net (Lewis,Mitchell T.) Date: Sun, 18 Sep 2016 20:59:43 -0700 (PDT) Subject: Huawei NE Message-ID: <1455834561.28949078.1474257583795.JavaMail.zimbra@techcompute.net> Hi All, Does anyone have any experiences with the Huawei NE platform in a service provider environment they can share? Private message is fine. I am comparing against Cisco ASR & Juniper MX. Regards, Mitchell T. Lewis MLewis at TechCompute.Net |203-816-0371 PGP Fingerprint: 79F2A12BAC77827581C734212AFA805732A1394E Public PGP Key From jason+nanog at lixfeld.ca Mon Sep 19 16:00:36 2016 From: jason+nanog at lixfeld.ca (Jason Lixfeld) Date: Mon, 19 Sep 2016 12:00:36 -0400 Subject: Customers announcing communities to SP of SP Message-ID: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> Hi, Consider the following scenario: - Customer A is a customer of SP A - SP A is a customer of SP B - SP B has a traffic engineering community implementation With regards to using BGP communities for TE: - Does SP A write their own community implementation that maps to (some portion of) the community implementation of SP B? - Does SP A write their own community implementation that has no mappings at all to the community implementation of SP B; any TE that is required to be pushed to SP B is done by some dialog and coordination between Customer A and SP A? - Does SP A allow Customer A to announce prefixes tagged with SP B?s communities[1][2] - Is this sort of thing really complicated today, but one of the goals of draft-heitz-idr-large-community? [1] Customer A has knowledge of SP A?s upstream SP B [2] This opens up a can of worms where SP A or SP B implements some communities prefixed with reserved ASes, so we?ll assume that SP A implements some method of allowing communities prefixed with ASes of SP A and SP B only. Thanks! From jcurran at arin.net Mon Sep 19 17:16:08 2016 From: jcurran at arin.net (John Curran) Date: Mon, 19 Sep 2016 17:16:08 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> Message-ID: <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> On Sep 14, 2016, at 4:59 PM, Christopher Morrow wrote: > > On Wed, Sep 14, 2016 at 4:04 PM, Bryan Fields wrote: > >> On 9/14/16 3:09 AM, Scott Weeks wrote: >>> >>> Yes, RPKI. That's what I was waiting for. Now we can get to >>> a real discussion >> ... >> sure as heck not going to sign a legally binding contract saying they do :) >> > don't have to... see press. Chris - To be clear, he would still end up bound to an agreement which provides that they indemnify the RIR regarding their RPKI usage (which was the complaint expressed in the NANOG community regarding ARIN?s RPKI terms and conditions) - "The Member shall indemnify the RIPE NCC against any and all third party claims filed against the RIPE NCC in relation to the Member?s use of the RIPE NCC Certification Service or the Certificate.? FYI, /John John Curran President and CEO ARIN From nanog at ics-il.net Mon Sep 19 17:34:48 2016 From: nanog at ics-il.net (Mike Hammett) Date: Mon, 19 Sep 2016 12:34:48 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <1684510435.11522.1474305367994.JavaMail.mhammett@ThunderFuck> Message-ID: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From rsk at gsp.org Mon Sep 19 17:53:24 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Mon, 19 Sep 2016 13:53:24 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <87k2e91fch.fsf@mid.deneb.enyo.de> References: <20160916131246.GP29651@dilbert.slimey.org> <20160918130703.GA17415@gsp.org> <87k2e91fch.fsf@mid.deneb.enyo.de> Message-ID: <20160919175324.GA27679@gsp.org> On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote: > * Rich Kulawiec: > > > For example: if the average number of outbound SSH connections > > established per hour per host across all hosts behind CGNAT is 3.2, > > and you see a host making 1100/hour: that's a problem. It might be > > someone who botched a Perl script; or it might be a botted host > > trying to brute-force its way into something. > > If you do this, you break Github. 1. I didn't know that: *how* does this break Github? 2. This is just an *example* of how to use the technique. It's not meant to be literal. The general approach of determining the statistical characteristics of "normal" and then flagging things that are "way outside normal" works -- but of course it requires sufficient knowledge to account for things like Github usage and/or infrequent events and/or usage spikes triggered by real-world events, etc. The more you do it, and the longer you do it, the better you'll get at it. (But of course the false positive rate will never be zero. That's why the question of what to do when anomalies happen isn't easy: poke a human? throttle? block? further analysis?) ---rsk From fw at deneb.enyo.de Mon Sep 19 19:55:56 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Mon, 19 Sep 2016 21:55:56 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses References: <20160916131246.GP29651@dilbert.slimey.org> <20160918130703.GA17415@gsp.org> <87k2e91fch.fsf@mid.deneb.enyo.de> <20160919175324.GA27679@gsp.org> Message-ID: <87mvj3slyr.fsf@mid.deneb.enyo.de> * Rich Kulawiec: > On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote: >> * Rich Kulawiec: >> >> > For example: if the average number of outbound SSH connections >> > established per hour per host across all hosts behind CGNAT is 3.2, >> > and you see a host making 1100/hour: that's a problem. It might be >> > someone who botched a Perl script; or it might be a botted host >> > trying to brute-force its way into something. >> >> If you do this, you break Github. > > 1. I didn't know that: *how* does this break Github? Github users create several orders of magnitude more SSH connections than average users because the most convenient way to set up read/write access is to use SSH. Depending on how you use Github, you might update lots and lots of local repositories from Github at certain times of the day. > 2. This is just an *example* of how to use the technique. It's not > meant to be literal. The general approach of determining the statistical > characteristics of "normal" and then flagging things that are "way > outside normal" works -- but of course it requires sufficient knowledge > to account for things like Github usage and/or infrequent events and/or > usage spikes triggered by real-world events, etc. Sure, and people already do this, and are not very flexible about it. Support staff isn't briefed, and claim they do such stochastic behavior adjustment across all (server) products, which I find difficult to believe. I'm worried that this leads to a future where tunnelling everything over HTTP(S) is no longer sufficient. You have to make it look like a web server or browser, too. Everything else risks triggering automated countermeasures. That's the anti-thesis of good protocol design. From jared at puck.nether.net Mon Sep 19 20:19:09 2016 From: jared at puck.nether.net (Jared Mauch) Date: Mon, 19 Sep 2016 16:19:09 -0400 Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: > On Sep 19, 2016, at 1:34 PM, Mike Hammett wrote: > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas I think the growing gap between those with high speed links and so-called slower links will be an ongoing issue. I?ve heard of tricks that the SP can do to avoid super large windows from occurring, but with the increased focus on tcp fast open and this data stuffing, I would expect the impact for these less studied low-speed links to get worse. I?ve been helping a local WISP prepare their fiber OSP installation to try and mitigate some of the problems they have with local capacity and to work around the worsening NLOS situations that occur with annual tree growth. - Jared From theodore at ciscodude.net Mon Sep 19 20:28:50 2016 From: theodore at ciscodude.net (Theodore Baschak) Date: Mon, 19 Sep 2016 15:28:50 -0500 Subject: Customers announcing communities to SP of SP In-Reply-To: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> References: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> Message-ID: In a previous $dayjob at a different ASN I was customers of a large-ish regional Canadian carrier (at 100M), and also of a small local guy (at 8M) with only Cogent upstream. I would prepend out the local guy 3x, and then I also tagged 174:3003 to have cogent prepend 3x more. This worked somewhat OK to make up for the inbalance in speeds. This type of use case is supported by, and works well with draft-heitz-idr-large-community. As an operator of a 32-bit ASN I have no ability to use communities with my ASN in them like 16-bit ASN operator could, and I have expressed so on the IETF IDR list. Theodore Baschak - AS395089 - Hextet Systems https://ciscodude.net/ - https://hextet.systems/ http://mbix.ca/ On Mon, Sep 19, 2016 at 11:00 AM, Jason Lixfeld wrote: > Hi, > > Consider the following scenario: > > - Customer A is a customer of SP A > - SP A is a customer of SP B > - SP B has a traffic engineering community implementation > > With regards to using BGP communities for TE: > > - Does SP A write their own community implementation that maps to (some > portion of) the community implementation of SP B? > - Does SP A write their own community implementation that has no mappings > at all to the community implementation of SP B; any TE that is required to > be pushed to SP B is done by some dialog and coordination between Customer > A and SP A? > - Does SP A allow Customer A to announce prefixes tagged with SP B?s > communities[1][2] > - Is this sort of thing really complicated today, but one of the goals of > draft-heitz-idr-large-community? > > [1] Customer A has knowledge of SP A?s upstream SP B > [2] This opens up a can of worms where SP A or SP B implements some > communities prefixed with reserved ASes, so we?ll assume that SP A > implements some method of allowing communities prefixed with ASes of SP A > and SP B only. > > Thanks! From martijnschmidt at i3d.net Mon Sep 19 21:23:56 2016 From: martijnschmidt at i3d.net (i3D.net - Martijn Schmidt) Date: Mon, 19 Sep 2016 23:23:56 +0200 Subject: Customers announcing communities to SP of SP In-Reply-To: References: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> Message-ID: Hi Jason, The following reply which I sent to the IDR mailing list might also be helpful for you to understand the way most of these designs currently work - as well as some of the problems we encounter with the existing RFC1997 communities: http://www.ietf.org/mail-archive/web/idr/current/msg16219.html draft-heitz-idr-large-community should tackle the missing 32-bit ASN feature, but should also resolve the private AS overlapping problem you're describing in [2] since it's a 32:32:32 format rather than 16:16. Best regards, Martijn Schmidt On 09/19/2016 10:28 PM, Theodore Baschak wrote: > In a previous $dayjob at a different ASN I was customers of a large-ish > regional Canadian carrier (at 100M), and also of a small local guy (at 8M) > with only Cogent upstream. I would prepend out the local guy 3x, and then I > also tagged 174:3003 to have cogent prepend 3x more. This worked somewhat > OK to make up for the inbalance in speeds. > > This type of use case is supported by, and works well with > draft-heitz-idr-large-community. As an operator of a 32-bit ASN I have no > ability to use communities with my ASN in them like 16-bit ASN operator > could, and I have expressed so on the IETF IDR list. > > > > Theodore Baschak - AS395089 - Hextet Systems > https://ciscodude.net/ - https://hextet.systems/ > http://mbix.ca/ > > > On Mon, Sep 19, 2016 at 11:00 AM, Jason Lixfeld > wrote: > >> Hi, >> >> Consider the following scenario: >> >> - Customer A is a customer of SP A >> - SP A is a customer of SP B >> - SP B has a traffic engineering community implementation >> >> With regards to using BGP communities for TE: >> >> - Does SP A write their own community implementation that maps to (some >> portion of) the community implementation of SP B? >> - Does SP A write their own community implementation that has no mappings >> at all to the community implementation of SP B; any TE that is required to >> be pushed to SP B is done by some dialog and coordination between Customer >> A and SP A? >> - Does SP A allow Customer A to announce prefixes tagged with SP B?s >> communities[1][2] >> - Is this sort of thing really complicated today, but one of the goals of >> draft-heitz-idr-large-community? >> >> [1] Customer A has knowledge of SP A?s upstream SP B >> [2] This opens up a can of worms where SP A or SP B implements some >> communities prefixed with reserved ASes, so we?ll assume that SP A >> implements some method of allowing communities prefixed with ASes of SP A >> and SP B only. >> >> Thanks! From jlewis at lewis.org Tue Sep 20 01:49:59 2016 From: jlewis at lewis.org (Jon Lewis) Date: Mon, 19 Sep 2016 21:49:59 -0400 (EDT) Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: On Mon, 19 Sep 2016, Mike Hammett wrote: > The principal complaint is that upstream of whatever is doing the rate > limiting for a given customer there is significantly more capacity being > utilized than the customer has purchased. This could happen briefly as > TCP adjusts to the capacity limitation, but in some situations this has > persisted for days at a time. I'll list out a few situations as best as > I can recall them. Some of these may even be merges of a couple > situations. The point is to show the general issue and develop a better > process for collecting what exactly is happening at the time and how to > address it. > > One situation had approximately 45 megabit/s of capacity being used up > by a customer that had a 1.5 megabit/s plan. All other traffic normally > held itself within the 1.5 megabit/s, but this particular CDN sent > excessively more for extended periods of time. It sounds like either the rate-limiting just isn't working, or the CDNs are trying too hard to ramp up the transfer rate in spite of your dropping some/most of the packets. I assume drops are happening either as part of the rate-limiting/policing, or simply as a result of trying to stuff 45mbit/s onto a 1.5mbit/s pipe....96.5% packet loss...and they're not slowing down at the sender?!? This is kind of a funny problem though, because CDNs get paid to deliver data, and they get compared/graded according to who can deliver the bits the fastest...and here you are complaining that they're delivering the bits too fast (or at least faster than you'd like them to). ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ From nanog at ics-il.net Tue Sep 20 03:05:31 2016 From: nanog at ics-il.net (Mike Hammett) Date: Mon, 19 Sep 2016 22:05:31 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <1128721862.12252.1474340730438.JavaMail.mhammett@ThunderFuck> http://www.theregister.co.uk/2016/06/08/is_win_10_ignoring_sysadmins_qos_settings/ This explains the recent situations (well, not really an explanation, but a bit more information from other people). Not so much for the ones going back a year or two. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" To: "NANOG" Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From morrowc.lists at gmail.com Tue Sep 20 03:58:32 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Mon, 19 Sep 2016 23:58:32 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> Message-ID: (caution! I don't really think arin is evil!) On Mon, Sep 19, 2016 at 1:16 PM, John Curran wrote: > On Sep 14, 2016, at 4:59 PM, Christopher Morrow > wrote: > > > > On Wed, Sep 14, 2016 at 4:04 PM, Bryan Fields > wrote: > > > >> On 9/14/16 3:09 AM, Scott Weeks wrote: > >>> > >>> Yes, RPKI. That's what I was waiting for. Now we can get to > >>> a real discussion > >> ... > >> sure as heck not going to sign a legally binding contract saying they > do :) > >> > > don't have to... see press. > > Chris - > > To be clear, he would still end up bound to an agreement which provides > that they > indemnify the RIR regarding their RPKI usage (which was the complaint > expressed > in the NANOG community regarding ARIN?s RPKI terms and conditions) - > > maybe, but his point was that the evil (evile?) arin would not be putting their clutches on his ip-address-spaces... Sure he's trading ARIN for RIPE here, but I diidn't think the RPA bit was his concern as much as the LRSA and 'now that you agree these are ip blocks are subject to the legacy registry services agreement, we (arin - with twisty mustasche) might decide to wrest them away from you!!!' I may have misinterpreted his issue, or not completed his unwritten thought about the RPA. legal/ripe-ncc-certification-service-terms-and-conditions> > "The Member shall indemnify the RIPE NCC against any and all third party > claims filed against the RIPE NCC in relation to the Member?s use of the > RIPE NCC Certification Service or the Certificate.? > > FYI, > /John > > John Curran > President and CEO > ARIN > > > > > > From george at cbcast.com Tue Sep 20 06:14:26 2016 From: george at cbcast.com (George Skorup) Date: Tue, 20 Sep 2016 01:14:26 -0500 Subject: CDN Overload? In-Reply-To: <1128721862.12252.1474340730438.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1128721862.12252.1474340730438.JavaMail.mhammett@ThunderFuck> Message-ID: I have witnessed this issue first hand for several years. Four for sure, maybe five or six. The very first one I remember is a customer doing Usenet downloads and using what he called an "internet download manager" which I assumed was screwing with TCP ACKs. I believe he was a 4Mbps user at the time and this download manager thing was causing 2 to maybe 2.5x his subscribed rate, as Mike says, on the upstream facing router interface. He shut down or uninstalled the software and it stopped. Yes, this customer is on PTMP fixed wireless. Traffic policing was taking place via MikroTik simple queue at the site router.. I could cut his downstream rate in half and it would follow with double still hitting the backhaul. I could also move his queue all the way to the border router and it was still there at double rate. BTW, we still have this guy as a customer on fixed wireless. He's been on 25/5Mbps for over a year. And we're about to upgrade him to 50/10Mbps with new gear. 25/5 and 50/10 is a far cry from this claimed "slow" WISP service. This shit ain't cheap to get to bumfsck Illinois so farmer Joe can watch porn and his kids can watch Netflix at the same time. Yup, we have slow NLOS service too, because customers decide they want the rural life buried in a mile of trees while "needing" the city benefits. If you want the gigabits, then move outta the sticks. Running a hundred combined miles of fiber to get to 20 customers that want to pay less than $50/mo is not feasible. /rant off Another time, maybe three years ago, we had a customer on Canopy 5.7 FSK at 4/1Mbps using the built-in QoS. He was watching Netflix and I saw 8Mbps hitting the AP's ethernet interface. I thought the Canopy scheduler was broken. Until I looked deeper and saw that it was working exactly as designed.. with 50% discard rate on his VC. I want to say this was from LLNW at the time. I could be totally wrong about that, I really don't remember. Now lets move the Windows 10 updates. A 'buried in the sticks' customer on Canopy 900 FSK. 1.5Mbps/384k. Multiple streams from Microsoft and LLNW at the same time. LLNW alone had maybe 10 streams going and was sending at over 15Mbps on average and at worst about 25Mbps... to a 1.5Mbps subscriber. I could throw in a MikroTik queue upstream which only moved the problem as that 15-25Mbps was still hitting backhaul links. And when I have a 100Mbps link going into the site, 25Mbps is a lot. We've had numerous customers call in for the last month or two with 'teh innernets is down, my phoen wyfy don't work either'. No, your Windows 10 updates are overloading your service. Shut off your PC to use your internet service. Telling a customer those exact words is ridiculous, but we have to do it. We had a known issue with a particular licensed microwave vendor's radios that we have in use. It was the ethernet buffer becoming saturated at nowhere near the RF link capacity. They put out a new software release and that was resolved. And that was well before this Windows 10 update overload stuff started. Normal TCP congestion control behavior works perfectly fine. It's not the network. It's the sender not doing normal TCP stuffs. I don't know why the CDNs and/or Microsoft thinks this is a good idea, but to me, it looks like a DDoS. I'm on some of the same lists as Mike and we know of many others reporting similar issues. A couple to the tune of 50-100Mbps overload destined for 5 or 10Mbps tier subscribers. So thanks to Mike for trying to get a conversation going on this topic. And it's not just us red headed step children WISPs. On 9/19/2016 10:05 PM, Mike Hammett wrote: > http://www.theregister.co.uk/2016/06/08/is_win_10_ignoring_sysadmins_qos_settings/ > > This explains the recent situations (well, not really an explanation, but a bit more information from other people). Not so much for the ones going back a year or two. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" > To: "NANOG" > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > From matthew at walster.org Tue Sep 20 07:44:24 2016 From: matthew at walster.org (Matthew Walster) Date: Tue, 20 Sep 2016 08:44:24 +0100 Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1128721862.12252.1474340730438.JavaMail.mhammett@ThunderFuck> Message-ID: On 20 Sep 2016 9:14 am, "George Skorup" wrote: > > Now lets move the Windows 10 updates. A 'buried in the sticks' customer on Canopy 900 FSK. 1.5Mbps/384k. Multiple streams from Microsoft and LLNW at the same time. LLNW alone had maybe 10 streams going and was sending at over 15Mbps on average and at worst about 25Mbps... to a 1.5Mbps subscriber. I could throw in a MikroTik queue upstream which only moved the problem as that 15-25Mbps was still hitting backhaul links. And when I have a 100Mbps link going into the site, 25Mbps is a lot. Maybe I'm being naive but this sounds like an issue primarily with buffers. Police rather than shape the traffic, and reduce the burst size, and a lot of this should disappear... M From nanog at ics-il.net Tue Sep 20 11:45:25 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 20 Sep 2016 06:45:25 -0500 (CDT) Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1128721862.12252.1474340730438.JavaMail.mhammett@ThunderFuck> Message-ID: <1075528304.12407.1474371920657.JavaMail.mhammett@ThunderFuck> What do most broadband platforms do for rate limiting? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Matthew Walster" To: "George Skorup" Cc: "nanog list" Sent: Tuesday, September 20, 2016 2:44:24 AM Subject: Re: CDN Overload? On 20 Sep 2016 9:14 am, "George Skorup" wrote: > > Now lets move the Windows 10 updates. A 'buried in the sticks' customer on Canopy 900 FSK. 1.5Mbps/384k. Multiple streams from Microsoft and LLNW at the same time. LLNW alone had maybe 10 streams going and was sending at over 15Mbps on average and at worst about 25Mbps... to a 1.5Mbps subscriber. I could throw in a MikroTik queue upstream which only moved the problem as that 15-25Mbps was still hitting backhaul links. And when I have a 100Mbps link going into the site, 25Mbps is a lot. Maybe I'm being naive but this sounds like an issue primarily with buffers. Police rather than shape the traffic, and reduce the burst size, and a lot of this should disappear... M From jcurran at arin.net Tue Sep 20 12:05:36 2016 From: jcurran at arin.net (John Curran) Date: Tue, 20 Sep 2016 12:05:36 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> Message-ID: <4B48ECB7-5878-4F8D-8E9E-0B010A6D8F43@arin.net> On Sep 19, 2016, at 11:58 PM, Christopher Morrow wrote: > > (caution! I don't really think arin is evil!) Nor do I? (but I will remind folks that organizations evolve based on participation, so ongoing diligence and involvement is definitely warranted.) > On Mon, Sep 19, 2016 at 1:16 PM, John Curran wrote: > >> To be clear, he would still end up bound to an agreement which provides >> that they >> indemnify the RIR regarding their RPKI usage (which was the complaint >> expressed >> in the NANOG community regarding ARIN?s RPKI terms and conditions) - >> >> > maybe, but his point was that the evil (evile?) arin would not be putting > their clutches on his ip-address-spaces... Sure he's trading ARIN for RIPE > here, but I diidn't think the RPA bit was his concern as much as the LRSA > and 'now that you agree these are ip blocks are subject to the legacy > registry services agreement, we (arin - with twisty mustasche) might decide > to wrest them away from you!!!? A distinct possibly, but much improved in the current LRSA (and RSA, which are the same document at this point.) Unless he?s planning to not pay the annual maintenance fee and ignore the notices and letters that follow over the next 120 days, or if going to make a serious misrepresentation in the process of claiming the rights to the address block, he?s fairly safe... for example, ARIN now specifically disclaims revocation for lack of utilization. (Furthermore, if ARIN breaches its obligations, the status of the address block reverts to the same prior to entry the LRSA ? this is definitely less than RIPE provides, which is effectively exit at any time, but far better than the original LRSA.) If you want to just use your legacy address block (wth the same services that where in place at ARIN?s formation), then you don?t need to enter into an LRSA ? but please do still update your registration in the ARIN registry to have current contact data, as this helps deter potential hijackers. If you want to have those services that were developed since ARIN?s formation, then I?d suggest reviewing the actual current LRSA agreement, as it is significantly improved over earlier versions. Thanks! /John John Curran President and CEO [Evil?] ARIN From theghost101 at gmail.com Tue Sep 20 12:24:32 2016 From: theghost101 at gmail.com (Danijel Starman) Date: Tue, 20 Sep 2016 14:24:32 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: Something similar happened to a local FantasyConon I was helping set up, we had only two PS4 machines there and accounts provided by Blizzard for Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in about 8h. There was no other traffic other then those two accounts playing Overwatch so my guess is that they have some too aggressive checks. I've managed to convince our ISP there to change the outside IP of the link so we got them working the next day but it happened again in 8h. -- *blap* On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart wrote: > All, > > We operate an access network with several hundred thousand users. > Increasingly > we're putting the users behind CGNAT in order to continue to give them an > IPv4 > service (we're all dual-stack, so they all get public IPv6 too). Due to the > demographic of our users, many of them are gamers. > > We're hitting a problem with PlayStationNetwork 'randomly' blocking some > of our > CGNAT outside addresses, because they claim to have received anomalous, or > 'attack' traffic from that IP. This obviously causes problems for the other > legitimate users who end up behind the same public IPv4 address. > > Despite numerous attempts to engage with PSN, they are unwilling to give us > any additional information which would allow us to identify the 'rogue' > users > on our network, or to identify the 'unwanted' traffic so that we could > either > block it, or use it to identify the rogue users ourselves. > > Has anyone else come up against the problem, and/or have any suggestions on > how best to resolve it? > > Many thanks in advance, > > Simon > > From nanog-post at rsuc.gweep.net Tue Sep 20 13:06:46 2016 From: nanog-post at rsuc.gweep.net (Joe Provo) Date: Tue, 20 Sep 2016 09:06:46 -0400 Subject: Customers announcing communities to SP of SP In-Reply-To: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> References: <3BB431CB-294D-46C2-A5DE-056F4A3C000A@lixfeld.ca> Message-ID: <20160920130646.GA58115@gweep.net> On Mon, Sep 19, 2016 at 12:00:36PM -0400, Jason Lixfeld wrote: > Hi, > > Consider the following scenario: > > - Customer A is a customer of SP A > - SP A is a customer of SP B > - SP B has a traffic engineering community implementation > > With regards to using BGP communities for TE: > > - Does SP A write their own community implementation that maps to (some portion of) the community implementation of SP B? > - Does SP A write their own community implementation that has no mappings at all to the community implementation of SP B; any TE that is required to be pushed to SP B is done by some dialog and coordination between Customer A and SP A? > - Does SP A allow Customer A to announce prefixes tagged with SP B???s communities[1][2] "Sometimes" for all of the above; it depends on the network. There are networs which strip all signalling but their own. There are those who will strip signalling to their immediate neighbors (expecting customers to use thri own). There are some which propagate anything and everything. IME, it is generally good to sanitize an input stream of signalling you use to reduce unknown/difficult to trace conditions. It is also a good principle for a sender to be aware of how far their dollars/euros/quatloos propagate, as that's pretty much the limit of guarantee others will act on their requests. In your scenario, when SP B changes their communities, they have no obligation (nor method) to let a downstream of a downstream know about it... > - Is this sort of thing really complicated today, but one of the > goals of draft-heitz-idr-large-community? It can be complicated - review the various way folks have published their policies via the compilation up at https://onestep.net/communities/ and you'll see a number of approaches. I can't speak for the authors, but by my reading draft-heitz-idr-large-community provides two things: - parity for 32b and 16b use in communities - the room to clearly express multiple party ASNs distinctly from 'take action' data, which we do not have now Cheers! Joe -- RSUC / GweepNet / Spunk / FnB / CotSG / Usenix / NANOG From morrowc.lists at gmail.com Tue Sep 20 14:48:45 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Tue, 20 Sep 2016 10:48:45 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <4B48ECB7-5878-4F8D-8E9E-0B010A6D8F43@arin.net> References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> <4B48ECB7-5878-4F8D-8E9E-0B010A6D8F43@arin.net> Message-ID: On Tue, Sep 20, 2016 at 8:05 AM, John Curran wrote: > On Sep 19, 2016, at 11:58 PM, Christopher Morrow > wrote: > > > > (caution! I don't really think arin is evil!) > > Nor do I? (but I will remind folks that organizations evolve based on > participation, > so ongoing diligence and involvement is definitely warranted.) > > > On Mon, Sep 19, 2016 at 1:16 PM, John Curran wrote: > > > >> To be clear, he would still end up bound to an agreement which provides > >> that they > >> indemnify the RIR regarding their RPKI usage (which was the complaint > >> expressed > >> in the NANOG community regarding ARIN?s RPKI terms and conditions) - > >> > >> > > maybe, but his point was that the evil (evile?) arin would not be putting > > their clutches on his ip-address-spaces... Sure he's trading ARIN for > RIPE > > here, but I diidn't think the RPA bit was his concern as much as the LRSA > > and 'now that you agree these are ip blocks are subject to the legacy > > registry services agreement, we (arin - with twisty mustasche) might > decide > > to wrest them away from you!!!? > > A distinct possibly, but much improved in the current LRSA (and RSA, which > are the same document at this point.) Unless he?s planning to not pay the > annual maintenance fee and ignore the notices and letters that follow over > the next 120 days, or if going to make a serious misrepresentation in the > process of claiming the rights to the address block, he?s fairly safe... > for > example, ARIN now specifically disclaims revocation for lack of > utilization. > (Furthermore, if ARIN breaches its obligations, the status of the address > block reverts to the same prior to entry the LRSA ? this is definitely less > than RIPE provides, which is effectively exit at any time, but far better > than > the original LRSA.) > > If you want to just use your legacy address block (wth the same services > that > where in place at ARIN?s formation), then you don?t need to enter into an > LRSA ? > but please do still update your registration in the ARIN registry to have > current > contact data, as this helps deter potential hijackers. If you want to > have those > services that were developed since ARIN?s formation, then I?d suggest > reviewing > the actual current LRSA agreement, as it is significantly improved over > earlier > versions. > > and probably: "If you think there are still improvements, show up at arin meetings and lobby for change" yes? > Thanks! > /John > > John Curran > President and CEO > [Evil?] ARIN > > > > From jcurran at arin.net Tue Sep 20 15:11:56 2016 From: jcurran at arin.net (John Curran) Date: Tue, 20 Sep 2016 15:11:56 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <20160914000951.CCA9D8C7@m0086238.ppops.net> <9BD92FCD-A8D4-47D9-B1D2-F92C9A8769D7@arin.net> <4B48ECB7-5878-4F8D-8E9E-0B010A6D8F43@arin.net>, Message-ID: <89E016CF-F063-487A-B6F9-11AE118C0BFB@arin.net> On Sep 20, 2016, at 10:48 AM, Christopher Morrow > wrote: On Tue, Sep 20, 2016 at 8:05 AM, John Curran > wrote: ... If you want to just use your legacy address block (wth the same services that where in place at ARIN's formation), then you don't need to enter into an LRSA - but please do still update your registration in the ARIN registry to have current contact data, as this helps deter potential hijackers. If you want to have those services that were developed since ARIN's formation, then I'd suggest reviewing the actual current LRSA agreement, as it is significantly improved over earlier versions. and probably: "If you think there are still improvements, show up at arin meetings and lobby for change" yes? Absolutely. /John From lists at mtin.net Tue Sep 20 15:33:17 2016 From: lists at mtin.net (Justin Wilson) Date: Tue, 20 Sep 2016 11:33:17 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible. Justin Wilson j2sw at mtin.net --- http://www.mtin.net Owner/CEO xISP Solutions- Consulting ? Data Centers - Bandwidth http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric > On Sep 20, 2016, at 8:24 AM, Danijel Starman wrote: > > Something similar happened to a local FantasyConon I was helping set up, we > had only two PS4 machines there and accounts provided by Blizzard for > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > about 8h. There was no other traffic other then those two accounts playing > Overwatch so my guess is that they have some too aggressive checks. I've > managed to convince our ISP there to change the outside IP of the link so > we got them working the next day but it happened again in 8h. > > -- > *blap* > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart wrote: > >> All, >> >> We operate an access network with several hundred thousand users. >> Increasingly >> we're putting the users behind CGNAT in order to continue to give them an >> IPv4 >> service (we're all dual-stack, so they all get public IPv6 too). Due to the >> demographic of our users, many of them are gamers. >> >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some >> of our >> CGNAT outside addresses, because they claim to have received anomalous, or >> 'attack' traffic from that IP. This obviously causes problems for the other >> legitimate users who end up behind the same public IPv4 address. >> >> Despite numerous attempts to engage with PSN, they are unwilling to give us >> any additional information which would allow us to identify the 'rogue' >> users >> on our network, or to identify the 'unwanted' traffic so that we could >> either >> block it, or use it to identify the rogue users ourselves. >> >> Has anyone else come up against the problem, and/or have any suggestions on >> how best to resolve it? >> >> Many thanks in advance, >> >> Simon >> >> > From nanog at ics-il.net Tue Sep 20 19:18:14 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 20 Sep 2016 14:18:14 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <450994741.13184.1474399089975.JavaMail.mhammett@ThunderFuck> This is what I'm asking of them: ===== Have you seen a CDN overloading a customer? Help me gather information on the issue. What CDN? What have you identified the traffic to be? What is the access network? Where is the rate limiting done? How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)? What is doing the rate limiting? What is the rate-limit set to? Upstream of the rate-limiter, what are you seeing for inbound traffic? One connection or many? How much traffic? How does other traffic behave when exceeding the rate limit? Where is NAT performed? What is doing NAT? Shared NAT or isolated to that customer? Have you done a packet capture before and after the rate limiter? The NAT device? Would you be willing to send a filtered packet capture (only the frames that relate to this CDN) to the CDN if they want it? There have been reports of CDNs sending more traffic than the customer can handle and ignores TCP convention to slow down. Trying to investigate this thoroughly so we can get the CDN to fix their system. Multiple CDNs have been shown to do this. ===== ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" To: "NANOG" Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From fw at deneb.enyo.de Tue Sep 20 20:34:57 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 20 Sep 2016 22:34:57 +0200 Subject: CDN Overload? In-Reply-To: (Jon Lewis's message of "Mon, 19 Sep 2016 21:49:59 -0400 (EDT)") References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <87twdamhse.fsf@mid.deneb.enyo.de> * Jon Lewis: > This is kind of a funny problem though, because CDNs get paid to > deliver data, and they get compared/graded according to who can > deliver the bits the fastest...and here you are complaining that > they're delivering the bits too fast (or at least faster than you'd > like them to). Surely CDNs bill packets which are subsequently dropped by the network. :) From marka at isc.org Wed Sep 21 01:29:49 2016 From: marka at isc.org (Mark Andrews) Date: Wed, 21 Sep 2016 11:29:49 +1000 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: Your message of "Tue, 20 Sep 2016 11:33:17 -0400." <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> References: <20160916131246.GP29651@dilbert.slimey.org> <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> Message-ID: <20160921012950.35D5E54B9721@rock.dv.isc.org> In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j2sw at mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From marka at isc.org Wed Sep 21 01:34:57 2016 From: marka at isc.org (Mark Andrews) Date: Wed, 21 Sep 2016 11:34:57 +1000 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: Your message of "Wed, 21 Sep 2016 11:29:49 +1000." Message-ID: <20160921013458.33DB054B97A4@rock.dv.isc.org> Mark Andrews writes: > > In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net>, Justin Wilson wri > te > s: > > PSN is one reason I am not a fan of CGNAT. All they see are tons of > > connections from the same IP. This results in them banning folks. Due > > to them being hacked so many times getting them to actually communicate > > is almost impossible. My .02 is just get the gamers a true public if at > > all possible. > > > > Justin Wilson > > j2sw at mtin.net > > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. Additionally is the a role for the SEC in ensuring that companies take IPv6 seriously? If I remember correctly they got involved with Y2K. Just because there isn't a hard date it doesn't mean that IPv6 is any less important than Y2K to your business's survival. > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From hugo at slabnet.com Wed Sep 21 01:46:01 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Tue, 20 Sep 2016 18:46:01 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> Message-ID: <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> Lucy, you got some (*serious*) 'splainin to do... http://research.dyn.com/2016/09/backconnects-suspicious-bgp-hijacks/ http://krebsonsecurity.com/2016/09/ddos-mitigation-firm-has-history-of-hijacks/ -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal On Sun 2016-Sep-18 22:25:44 -0400, Tom Beecher wrote: >So after reading your explanation of things... > >Your technical protections for your client proved sufficient to handle the >attack. You took OFFENSIVE action by hijacking the IP space. By your own >statements, it was only in response to threats against your company. You >were no longer providing DDoS protection to a client. You were exacting a >vendetta against someone who was being MEAN to you. Even if that person >probably deserved it, you still cannot do what was done. > >I appreciate the desire to want to protect friends and family from >anonymous threats, and also realize how ill equipped law enforcement >usually is while something like this is occurring. > >However, in my view, by taking the action you did, you have shown your >company isn't ready to be operating in the security space. Being threatened >by bad actors is a nominal part of doing business in the security space. >Unfortunately you didn't handle it well, and I think that will stick to you >for a long time. > >On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend >wrote: > >> @ca & Matt - No, we do not plan to ever intentionally perform a >> non-authorized BGP hijack in the future. >> >> @Steve - Correct, the attack had already been mitigated. The decision to >> hijack the attackers IP space was to deal with their threats, which if >> carried through could have potentially lead to physical harm. Although the >> hijack gave us a unique insight into the attackers services, it was not a >> factor that influenced my decision. >> >> @Blake & Mel - We will likely cover some of these questions in a future >> blog post. >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From mel at beckman.org Wed Sep 21 03:21:04 2016 From: mel at beckman.org (Mel Beckman) Date: Wed, 21 Sep 2016 03:21:04 +0000 Subject: "Defensive" BGP hijacking? In-Reply-To: <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> , <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> Message-ID: <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> While I was reading the krebsonsecurity.com article cited below, the site, hosted at Akamai address 72.52.7.144, became non responsive and now appears to be offline. Traceroutes stop before the Akamai-SWIPed border within Telia, as if blackholed (but adjacent IPs pass through to Akamai): traceroute to krebsonsecurity.com (72.52.7.144), 64 hops max, 40 byte packets 1 router1.sb.becknet.com (206.83.0.1) 0.771 ms 0.580 ms 0.342 ms 2 206-190-77-9.static.twtelecom.net (206.190.77.9) 0.715 ms 1.026 ms 0.744 ms 3 ae1-90g.ar7.lax1.gblx.net (67.17.75.18) 9.532 ms 6.567 ms 2.912 ms 4 ae10.edge1.losangeles9.level3.net (4.68.111.21) 2.919 ms 2.925 ms 2.904 ms 5 telia-level3-4x10g.losangeles.level3.net (4.68.70.130) 3.981 ms 3.567 ms 3.401 ms 6 sjo-b21-link.telia.net (62.115.116.40) 11.209 ms 11.140 ms 11.161 ms 7 * * * 8 * * * 9 * * * 10 * * * Weird coincidence? -mel beckman > On Sep 20, 2016, at 6:46 PM, Hugo Slabbert wrote: > > Lucy, you got some (*serious*) 'splainin to do... > > http://research.dyn.com/2016/09/backconnects-suspicious-bgp-hijacks/ > http://krebsonsecurity.com/2016/09/ddos-mitigation-firm-has-history-of-hijacks/ > > -- > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > pgp key: B178313E | also on Signal > >> On Sun 2016-Sep-18 22:25:44 -0400, Tom Beecher wrote: >> >> So after reading your explanation of things... >> >> Your technical protections for your client proved sufficient to handle the >> attack. You took OFFENSIVE action by hijacking the IP space. By your own >> statements, it was only in response to threats against your company. You >> were no longer providing DDoS protection to a client. You were exacting a >> vendetta against someone who was being MEAN to you. Even if that person >> probably deserved it, you still cannot do what was done. >> >> I appreciate the desire to want to protect friends and family from >> anonymous threats, and also realize how ill equipped law enforcement >> usually is while something like this is occurring. >> >> However, in my view, by taking the action you did, you have shown your >> company isn't ready to be operating in the security space. Being threatened >> by bad actors is a nominal part of doing business in the security space. >> Unfortunately you didn't handle it well, and I think that will stick to you >> for a long time. >> >> On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend >> wrote: >> >>> @ca & Matt - No, we do not plan to ever intentionally perform a >>> non-authorized BGP hijack in the future. >>> >>> @Steve - Correct, the attack had already been mitigated. The decision to >>> hijack the attackers IP space was to deal with their threats, which if >>> carried through could have potentially lead to physical harm. Although the >>> hijack gave us a unique insight into the attackers services, it was not a >>> factor that influenced my decision. >>> >>> @Blake & Mel - We will likely cover some of these questions in a future >>> blog post. >>> From justin at cloudflare.com Wed Sep 21 03:26:52 2016 From: justin at cloudflare.com (Justin Paine) Date: Tue, 20 Sep 2016 20:26:52 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> Message-ID: earlier on Twitter Krebs said he was hit by 665Gbps attack (so says Prolexic/Akamai). Could be ongoing/related. ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Tue, Sep 20, 2016 at 8:21 PM, Mel Beckman wrote: > While I was reading the krebsonsecurity.com article cited below, the site, hosted at Akamai address 72.52.7.144, became non responsive and now appears to be offline. Traceroutes stop before the Akamai-SWIPed border within Telia, as if blackholed (but adjacent IPs pass through to Akamai): > > traceroute to krebsonsecurity.com (72.52.7.144), 64 hops max, 40 byte packets > 1 router1.sb.becknet.com (206.83.0.1) 0.771 ms 0.580 ms 0.342 ms > 2 206-190-77-9.static.twtelecom.net (206.190.77.9) 0.715 ms 1.026 ms 0.744 ms > 3 ae1-90g.ar7.lax1.gblx.net (67.17.75.18) 9.532 ms 6.567 ms 2.912 ms > 4 ae10.edge1.losangeles9.level3.net (4.68.111.21) 2.919 ms 2.925 ms 2.904 ms > 5 telia-level3-4x10g.losangeles.level3.net (4.68.70.130) 3.981 ms 3.567 ms 3.401 ms > 6 sjo-b21-link.telia.net (62.115.116.40) 11.209 ms 11.140 ms 11.161 ms > 7 * * * > 8 * * * > 9 * * * > 10 * * * > > Weird coincidence? > > -mel beckman > >> On Sep 20, 2016, at 6:46 PM, Hugo Slabbert wrote: >> >> Lucy, you got some (*serious*) 'splainin to do... >> >> http://research.dyn.com/2016/09/backconnects-suspicious-bgp-hijacks/ >> http://krebsonsecurity.com/2016/09/ddos-mitigation-firm-has-history-of-hijacks/ >> >> -- >> Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com >> pgp key: B178313E | also on Signal >> >>> On Sun 2016-Sep-18 22:25:44 -0400, Tom Beecher wrote: >>> >>> So after reading your explanation of things... >>> >>> Your technical protections for your client proved sufficient to handle the >>> attack. You took OFFENSIVE action by hijacking the IP space. By your own >>> statements, it was only in response to threats against your company. You >>> were no longer providing DDoS protection to a client. You were exacting a >>> vendetta against someone who was being MEAN to you. Even if that person >>> probably deserved it, you still cannot do what was done. >>> >>> I appreciate the desire to want to protect friends and family from >>> anonymous threats, and also realize how ill equipped law enforcement >>> usually is while something like this is occurring. >>> >>> However, in my view, by taking the action you did, you have shown your >>> company isn't ready to be operating in the security space. Being threatened >>> by bad actors is a nominal part of doing business in the security space. >>> Unfortunately you didn't handle it well, and I think that will stick to you >>> for a long time. >>> >>> On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend >>> wrote: >>> >>>> @ca & Matt - No, we do not plan to ever intentionally perform a >>>> non-authorized BGP hijack in the future. >>>> >>>> @Steve - Correct, the attack had already been mitigated. The decision to >>>> hijack the attackers IP space was to deal with their threats, which if >>>> carried through could have potentially lead to physical harm. Although the >>>> hijack gave us a unique insight into the attackers services, it was not a >>>> factor that influenced my decision. >>>> >>>> @Blake & Mel - We will likely cover some of these questions in a future >>>> blog post. >>>> From beecher at beecher.cc Wed Sep 21 03:28:47 2016 From: beecher at beecher.cc (Tom Beecher) Date: Tue, 20 Sep 2016 23:28:47 -0400 Subject: "Defensive" BGP hijacking? In-Reply-To: <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> Message-ID: Brian Krebs tweeted out that Prolexic reported a 665Gbps attack directed at his site. https://twitter.com/briankrebs/status/778398865619836928 On Tue, Sep 20, 2016 at 11:21 PM, Mel Beckman wrote: > While I was reading the krebsonsecurity.com article cited below, the > site, hosted at Akamai address 72.52.7.144, became non responsive and now > appears to be offline. Traceroutes stop before the Akamai-SWIPed border > within Telia, as if blackholed (but adjacent IPs pass through to Akamai): > > traceroute to krebsonsecurity.com (72.52.7.144), 64 hops max, 40 byte > packets > 1 router1.sb.becknet.com (206.83.0.1) 0.771 ms 0.580 ms 0.342 ms > 2 206-190-77-9.static.twtelecom.net (206.190.77.9) 0.715 ms 1.026 ms > 0.744 ms > 3 ae1-90g.ar7.lax1.gblx.net (67.17.75.18) 9.532 ms 6.567 ms 2.912 ms > 4 ae10.edge1.losangeles9.level3.net (4.68.111.21) 2.919 ms 2.925 ms > 2.904 ms > 5 telia-level3-4x10g.losangeles.level3.net (4.68.70.130) 3.981 ms > 3.567 ms 3.401 ms > 6 sjo-b21-link.telia.net (62.115.116.40) 11.209 ms 11.140 ms 11.161 > ms > 7 * * * > 8 * * * > 9 * * * > 10 * * * > > Weird coincidence? > > -mel beckman > > > On Sep 20, 2016, at 6:46 PM, Hugo Slabbert wrote: > > > > Lucy, you got some (*serious*) 'splainin to do... > > > > http://research.dyn.com/2016/09/backconnects-suspicious-bgp-hijacks/ > > http://krebsonsecurity.com/2016/09/ddos-mitigation-firm- > has-history-of-hijacks/ > > > > -- > > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > > pgp key: B178313E | also on Signal > > > >> On Sun 2016-Sep-18 22:25:44 -0400, Tom Beecher > wrote: > >> > >> So after reading your explanation of things... > >> > >> Your technical protections for your client proved sufficient to handle > the > >> attack. You took OFFENSIVE action by hijacking the IP space. By your own > >> statements, it was only in response to threats against your company. You > >> were no longer providing DDoS protection to a client. You were exacting > a > >> vendetta against someone who was being MEAN to you. Even if that person > >> probably deserved it, you still cannot do what was done. > >> > >> I appreciate the desire to want to protect friends and family from > >> anonymous threats, and also realize how ill equipped law enforcement > >> usually is while something like this is occurring. > >> > >> However, in my view, by taking the action you did, you have shown your > >> company isn't ready to be operating in the security space. Being > threatened > >> by bad actors is a nominal part of doing business in the security space. > >> Unfortunately you didn't handle it well, and I think that will stick to > you > >> for a long time. > >> > >> On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend < > bryant at backconnect.com> > >> wrote: > >> > >>> @ca & Matt - No, we do not plan to ever intentionally perform a > >>> non-authorized BGP hijack in the future. > >>> > >>> @Steve - Correct, the attack had already been mitigated. The decision > to > >>> hijack the attackers IP space was to deal with their threats, which if > >>> carried through could have potentially lead to physical harm. Although > the > >>> hijack gave us a unique insight into the attackers services, it was > not a > >>> factor that influenced my decision. > >>> > >>> @Blake & Mel - We will likely cover some of these questions in a future > >>> blog post. > >>> > From Valdis.Kletnieks at vt.edu Wed Sep 21 04:27:42 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Wed, 21 Sep 2016 00:27:42 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160921012950.35D5E54B9721@rock.dv.isc.org> References: <20160916131246.GP29651@dilbert.slimey.org> <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> <20160921012950.35D5E54B9721@rock.dv.isc.org> Message-ID: <18954.1474432062@turing-police.cc.vt.edu> On Wed, 21 Sep 2016 11:29:49 +1000, Mark Andrews said: > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. Interestingly enough, the Playstation 4 has at least rudimentary IPv6 support - it will DHCPv6 and answer pings. Threw me for a loop first time I saw it, I couldn't figure out what unaccounted-for gear I had that was grabbing an IPv6 address... :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From bryant at backconnect.com Wed Sep 21 04:28:50 2016 From: bryant at backconnect.com (Bryant Townsend) Date: Tue, 20 Sep 2016 21:28:50 -0700 Subject: "Defensive" BGP hijacking? In-Reply-To: References: <366BECC3-551E-4C39-B037-65CE6945D6F3@blighty.com> <20160921014601.2njsjkzgy3jitdoi@slab-wks-04.int.slabnet.com> <3988530B-1624-4017-ADAB-E68DEBE12BDB@beckman.org> Message-ID: Hello, We wanted to clarify that we are not the ones behind these attacks and we were not the ones behind the previous hijackings. We have also been the targets of DDoS attacks reaching up to 200+ Gbps (~20 times a day), every day since Krebs' original article that included our name. We believe these attacks are coming from vDOS past customers and other botnets that used the vDOS service for launching and selling attacks. We have also been targeted with what seems to be multiple e-mail list bombs in attempts to delay our response time. As I mentioned before, NANOG's trust means everything in this industry and we want to be able to answer as much as we can. Sincerely, Bryant Townsend On Tue, Sep 20, 2016 at 8:28 PM, Tom Beecher wrote: > Brian Krebs tweeted out that Prolexic reported a 665Gbps attack directed at > his site. > > https://twitter.com/briankrebs/status/778398865619836928 > > On Tue, Sep 20, 2016 at 11:21 PM, Mel Beckman wrote: > > > While I was reading the krebsonsecurity.com article cited below, the > > site, hosted at Akamai address 72.52.7.144, became non responsive and now > > appears to be offline. Traceroutes stop before the Akamai-SWIPed border > > within Telia, as if blackholed (but adjacent IPs pass through to Akamai): > > > > traceroute to krebsonsecurity.com (72.52.7.144), 64 hops max, 40 byte > > packets > > 1 router1.sb.becknet.com (206.83.0.1) 0.771 ms 0.580 ms 0.342 ms > > 2 206-190-77-9.static.twtelecom.net (206.190.77.9) 0.715 ms 1.026 ms > > 0.744 ms > > 3 ae1-90g.ar7.lax1.gblx.net (67.17.75.18) 9.532 ms 6.567 ms 2.912 > ms > > 4 ae10.edge1.losangeles9.level3.net (4.68.111.21) 2.919 ms 2.925 ms > > 2.904 ms > > 5 telia-level3-4x10g.losangeles.level3.net (4.68.70.130) 3.981 ms > > 3.567 ms 3.401 ms > > 6 sjo-b21-link.telia.net (62.115.116.40) 11.209 ms 11.140 ms 11.161 > > ms > > 7 * * * > > 8 * * * > > 9 * * * > > 10 * * * > > > > Weird coincidence? > > > > -mel beckman > > > > > On Sep 20, 2016, at 6:46 PM, Hugo Slabbert wrote: > > > > > > Lucy, you got some (*serious*) 'splainin to do... > > > > > > http://research.dyn.com/2016/09/backconnects-suspicious-bgp-hijacks/ > > > http://krebsonsecurity.com/2016/09/ddos-mitigation-firm- > > has-history-of-hijacks/ > > > > > > -- > > > Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com > > > pgp key: B178313E | also on Signal > > > > > >> On Sun 2016-Sep-18 22:25:44 -0400, Tom Beecher > > wrote: > > >> > > >> So after reading your explanation of things... > > >> > > >> Your technical protections for your client proved sufficient to handle > > the > > >> attack. You took OFFENSIVE action by hijacking the IP space. By your > own > > >> statements, it was only in response to threats against your company. > You > > >> were no longer providing DDoS protection to a client. You were > exacting > > a > > >> vendetta against someone who was being MEAN to you. Even if that > person > > >> probably deserved it, you still cannot do what was done. > > >> > > >> I appreciate the desire to want to protect friends and family from > > >> anonymous threats, and also realize how ill equipped law enforcement > > >> usually is while something like this is occurring. > > >> > > >> However, in my view, by taking the action you did, you have shown your > > >> company isn't ready to be operating in the security space. Being > > threatened > > >> by bad actors is a nominal part of doing business in the security > space. > > >> Unfortunately you didn't handle it well, and I think that will stick > to > > you > > >> for a long time. > > >> > > >> On Tue, Sep 13, 2016 at 3:29 PM, Bryant Townsend < > > bryant at backconnect.com> > > >> wrote: > > >> > > >>> @ca & Matt - No, we do not plan to ever intentionally perform a > > >>> non-authorized BGP hijack in the future. > > >>> > > >>> @Steve - Correct, the attack had already been mitigated. The decision > > to > > >>> hijack the attackers IP space was to deal with their threats, which > if > > >>> carried through could have potentially lead to physical harm. > Although > > the > > >>> hijack gave us a unique insight into the attackers services, it was > > not a > > >>> factor that influenced my decision. > > >>> > > >>> @Blake & Mel - We will likely cover some of these questions in a > future > > >>> blog post. > > >>> > > > From swmike at swm.pp.se Wed Sep 21 05:56:43 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Wed, 21 Sep 2016 07:56:43 +0200 (CEST) Subject: importance of fiber cleaning Message-ID: https://www.sunet.se/blogg/long-read-cleanliness-is-a-virtue/ This is an excellent article regarding fiber cleaning and its importance. Please do share with other people in our business. I'm sure lack of proper fiber cleaning causes a lot of unneccessary outages and operational problems worldwide, partly because people aren't aware of its importance. -- Mikael Abrahamsson email: swmike at swm.pp.se From baldur.norddahl at gmail.com Wed Sep 21 08:37:44 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Wed, 21 Sep 2016 10:37:44 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <20160916131246.GP29651@dilbert.slimey.org> References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: Hi We have the opposite problem with PSN: Sometimes they will send abuse reports with several of our IP addresses listed. The problem with that is that we can not give data about one customer to another customer. By listing multiple IP addresses we are prevented from forwarding the email to the customer. Which means we may ignore it instead. Regards, Baldur From baldur.norddahl at gmail.com Wed Sep 21 08:58:52 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Wed, 21 Sep 2016 10:58:52 +0200 Subject: importance of fiber cleaning In-Reply-To: References: Message-ID: It is a good article. It is missing a few points: If you are going to do the full efford of cleaning and then microscope each connector, you would also want to finish off by doing a OTDR scan of the link. This is your documentation for a clean link. Always use optics that can monitor the signal level. The reality is that best practice, as described in the article, will not always be followed. In most case you will be good anyway as long your optics report back a signal strength with a good margin. Have your automated monitoring system watch over those signal levels. Slightly dirty connectors will often give a sufficient link quality anyway if you have plenty of power budget to spare. We use many 1G single mode BIDI optics which cost about 10 USD each for 20 km modules and most of the links are only 1-5 km. The customer end of those links are probably all half dirty, but nobody cares as long we get a strong signal back with power budget to spare. Regards, Baldur On 09/21/2016 07:56 AM, Mikael Abrahamsson wrote: > > https://www.sunet.se/blogg/long-read-cleanliness-is-a-virtue/ > > This is an excellent article regarding fiber cleaning and its > importance. Please do share with other people in our business. I'm > sure lack of proper fiber cleaning causes a lot of unneccessary > outages and operational problems worldwide, partly because people > aren't aware of its importance. > From baldur.norddahl at gmail.com Wed Sep 21 09:02:30 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Wed, 21 Sep 2016 11:02:30 +0200 Subject: CDN Overload? In-Reply-To: <87twdamhse.fsf@mid.deneb.enyo.de> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <87twdamhse.fsf@mid.deneb.enyo.de> Message-ID: <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> How come we have never seen this problem? We have a ton of DSL and many of those are slow, but no customer complaints about overloaded lines from CDN networks. Could it be that the way you throttle the bandwidth is defect? It is easy to blame the other guy but could it be that you are doing it wrong? Regards, Badur From rdobbins at arbor.net Wed Sep 21 09:14:01 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Wed, 21 Sep 2016 16:14:01 +0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <20160916131246.GP29651@dilbert.slimey.org> Message-ID: <2D48958B-1E24-4172-B515-D1368042AF91@arbor.net> On 21 Sep 2016, at 15:37, Baldur Norddahl wrote: > Which means we may ignore it instead. . . . copy/paste or awk/sed or whatever isn't an option? If not, have you requested a) separate notifications per source and/or b) a more textual-manipulation-friendly format? Unless they're sending .gifs or something, surely this might be possible, yes? It seems within the realm of possibility this sort of response - or lack thereof - could result in some gaming network operators becoming a bit jaded. And perhaps some customers, too. ----------------------------------- Roland Dobbins From mkamal at noor.net Wed Sep 21 09:14:40 2016 From: mkamal at noor.net (Mohamed Kamal) Date: Wed, 21 Sep 2016 11:14:40 +0200 Subject: Yahoo as#10310 reachability problem Message-ID: <384087a2-2c87-0bf4-fe6a-3144bd870379@noor.net> Can someone from Yahoo as#10310 contact me off-list, we have some problems reaching Yahoo through Telia and GTT. Thanks. -- Mohamed Kamal Core Network Sr. Engineer From mel at beckman.org Wed Sep 21 09:53:49 2016 From: mel at beckman.org (Mel Beckman) Date: Wed, 21 Sep 2016 09:53:49 +0000 Subject: importance of fiber cleaning In-Reply-To: References: Message-ID: <6767F84A-20E2-422D-A8F4-F0E6A522B1E0@beckman.org> This is a very comprehensive article, and worth handing out to techs. I have one comment on Balder?s OTDR suggestion, and one on the article?s microscope instructions. Although it certainly can?t hurt to run an OTDR test (except for extended downtime), I fear hauling out the extra gear will prompt many techs to put off fiber cleaning. In my experience, just doing the cleaning solves 99.9% of the problem. Anything that an OTDR would pick up would likely severely impact performance, while dirty connector will just increase the error rate. Also, the article didn?t mention eye safety when using a fiber microscope. The example showed a USB digital video microscope, but many maintainer kits in the field have much cheaper direct-view optical microscopes. Viewing an energized fiber with a direct-view microscope can cause major eye damage. I recommend all fiber kits throw out their optical scopes and substitute a USB or WiFI scope (some of these can be used with a cell phone or tablet). -mel > On Sep 21, 2016, at 1:58 AM, Baldur Norddahl wrote: > > It is a good article. It is missing a few points: > > If you are going to do the full efford of cleaning and then microscope each connector, you would also want to finish off by doing a OTDR scan of the link. This is your documentation for a clean link. > > Always use optics that can monitor the signal level. The reality is that best practice, as described in the article, will not always be followed. In most case you will be good anyway as long your optics report back a signal strength with a good margin. Have your automated monitoring system watch over those signal levels. > > Slightly dirty connectors will often give a sufficient link quality anyway if you have plenty of power budget to spare. We use many 1G single mode BIDI optics which cost about 10 USD each for 20 km modules and most of the links are only 1-5 km. The customer end of those links are probably all half dirty, but nobody cares as long we get a strong signal back with power budget to spare. > > Regards, > > Baldur > > On 09/21/2016 07:56 AM, Mikael Abrahamsson wrote: >> >> https://www.sunet.se/blogg/long-read-cleanliness-is-a-virtue/ >> >> This is an excellent article regarding fiber cleaning and its importance. Please do share with other people in our business. I'm sure lack of proper fiber cleaning causes a lot of unneccessary outages and operational problems worldwide, partly because people aren't aware of its importance. >> > From nanog at ics-il.net Wed Sep 21 11:45:38 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 21 Sep 2016 06:45:38 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <87twdamhse.fsf@mid.deneb.enyo.de> <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> Message-ID: <493052883.13531.1474458335351.JavaMail.mhammett@ThunderFuck> Likewise, why was it never an issue before and why does it only affect certain types of traffic from certain CDNs? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Baldur Norddahl" To: nanog at nanog.org Sent: Wednesday, September 21, 2016 4:02:30 AM Subject: Re: CDN Overload? How come we have never seen this problem? We have a ton of DSL and many of those are slow, but no customer complaints about overloaded lines from CDN networks. Could it be that the way you throttle the bandwidth is defect? It is easy to blame the other guy but could it be that you are doing it wrong? Regards, Badur From hugge at nordu.net Wed Sep 21 12:25:57 2016 From: hugge at nordu.net (=?UTF-8?Q?Fredrik_Korsb=c3=a4ck?=) Date: Wed, 21 Sep 2016 14:25:57 +0200 Subject: importance of fiber cleaning In-Reply-To: <6767F84A-20E2-422D-A8F4-F0E6A522B1E0@beckman.org> References: <6767F84A-20E2-422D-A8F4-F0E6A522B1E0@beckman.org> Message-ID: <78a3b301-dff5-7c88-0d8b-da7d416b11db@nordu.net> On 21/09/16 11:53, Mel Beckman wrote: > This is a very comprehensive article, and worth handing out to techs. I have one comment on Balder?s OTDR suggestion, and one on the article?s microscope instructions. > > Although it certainly can?t hurt to run an OTDR test (except for extended downtime), I fear hauling out the extra gear will prompt many techs to put off fiber cleaning. In my experience, just doing the cleaning solves 99.9% of the problem. Anything that an OTDR would pick up would likely severely impact performance, while dirty connector will just increase the error rate. > > Also, the article didn?t mention eye safety when using a fiber microscope. The example showed a USB digital video microscope, but many maintainer kits in the field have much cheaper direct-view optical microscopes. Viewing an energized fiber with a direct-view microscope can cause major eye damage. I recommend all fiber kits throw out their optical scopes and substitute a USB or WiFI scope (some of these can be used with a cell phone or tablet). > > -mel Cool that this article got posted here (im not the author, but im the dude in the pictures). One reason we didn't bring up OTDR in the article is that OTDR-test only works on certain occasions. If you have problems with a circuit and decide to take it down to clean it up and inspect patches and ODFs you need to also disconnect the other side if you feel doing a OTDR-test. Launching light from a OTDR-instrument where you have a regular transceiver on the other side gives a potential risk of destroying it. Also if running the OTDR towards a TX will also skew up the results since there is already light in the fibre. We have however (both me and j?rgen) done a few articles where we go through OTDR-stuff. https://www.sunet.se/blogg/sites-and-fibers-being-delivered/ https://www.sunet.se/blogg/first-test-on-real-fiber/ https://www.sunet.se/blogg/full-speed-ahead/ https://www.sunet.se/blogg/otdr-grundlaggande-om/ (use google translate on that one, i can get it translated if its of interest) -- hugge From josh at kyneticwifi.com Wed Sep 21 12:36:29 2016 From: josh at kyneticwifi.com (Josh Reynolds) Date: Wed, 21 Sep 2016 07:36:29 -0500 Subject: CDN Overload? In-Reply-To: <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <87twdamhse.fsf@mid.deneb.enyo.de> <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> Message-ID: With so many geographically diverse complaints on many hardware routing and switching platforms, I'm going to go with a "no". On Sep 21, 2016 4:04 AM, "Baldur Norddahl" wrote: > How come we have never seen this problem? We have a ton of DSL and many of > those are slow, but no customer complaints about overloaded lines from CDN > networks. > > Could it be that the way you throttle the bandwidth is defect? It is easy > to blame the other guy but could it be that you are doing it wrong? > > Regards, > > Badur > > From s.kakaroukas at connecticore.com Wed Sep 21 13:47:12 2016 From: s.kakaroukas at connecticore.com (Spyros Kakaroukas) Date: Wed, 21 Sep 2016 13:47:12 +0000 Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1684510435.11522.1474305367994.JavaMail.mhammett@ThunderFuck> <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <038E6A84-28BA-488B-8EF7-F69FFB5EA0FD@connecticore.com> That's interesting. Once, a few years/jobs/etc ago, I observed a flow from mobile youtube being really really bursty, peaking to a 40-50mbps on a 10mbps circuit, but that was the only time I've ever seen such an issue. After that one flow died, it never happened again. That aside, I do work for a business-focused mostly-wireless SP at the moment and I haven't had any issues with CDNs so far. The only similar incidents I can recall involved customers running programs like aspera and signiant which, when misconfigured, can result in quite some volume coming your way. My thoughts and words are my own. Spyros This e-mail and any attachment(s) contained within are confidential and are intended only for the use of the individual to whom they are addressed. The information contained in this communication may be privileged, or exempt from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender and delete the communication without retaining any copies. Connecticore SA is not responsible for, nor endorses, any opinion, recommendation, conclusion, solicitation, offer or agreement or any information contained in this communication. From nanog at ics-il.net Wed Sep 21 14:08:55 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 21 Sep 2016 09:08:55 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" To: "NANOG" Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From baldur.norddahl at gmail.com Wed Sep 21 14:32:58 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Wed, 21 Sep 2016 16:32:58 +0200 Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <87twdamhse.fsf@mid.deneb.enyo.de> <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> Message-ID: It appears all complaints are from SP doing wireless. I am going to go with a yes and put forth a these that these guys have a common factor somewhere. It could be equipment from a some popular vendor of wireless or maybe some common method to throttle that is popular in the wireless community. I note that while we have slow links we have no throttling or bandwidth management going on except for the buffering that happens in the DSLAM. Also there is no way to cheat. If you send 4 mbps to a 2 mbps DSL it will drop half of the traffic and TCP will not survive that. The CDN would have an effective transfer rate approaching zero for that customer. That seems to be a rather bad business proposal seen from the view if the CDN so they would not do that. The other customers will be unaffected as the DSLAM itself has plenty of capacity. Regards Baldur Den 21. sep. 2016 14.36 skrev "Josh Reynolds" : > With so many geographically diverse complaints on many hardware routing > and switching platforms, I'm going to go with a "no". > > On Sep 21, 2016 4:04 AM, "Baldur Norddahl" > wrote: > >> How come we have never seen this problem? We have a ton of DSL and many >> of those are slow, but no customer complaints about overloaded lines from >> CDN networks. >> >> Could it be that the way you throttle the bandwidth is defect? It is easy >> to blame the other guy but could it be that you are doing it wrong? >> >> Regards, >> >> Badur >> >> From nanog at ics-il.net Wed Sep 21 14:40:33 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 21 Sep 2016 09:40:33 -0500 (CDT) Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <87twdamhse.fsf@mid.deneb.enyo.de> <7bf5acad-e2c8-5545-c46e-874d51b4cc2b@gmail.com> Message-ID: <1467003575.13905.1474468828283.JavaMail.mhammett@ThunderFuck> I've had DSL and AE service providers respond with the issues. So far there is not a common element other than CDNs. That's the point of the questions I'm asking, to gather a ton of information and then figure out how to act on it. You're assuming that the CDNs are using an unmolested, vanilla TCP stack. That may not be the case, especially if doing something like Fast TCP. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Baldur Norddahl" To: nanog at nanog.org Sent: Wednesday, September 21, 2016 9:32:58 AM Subject: Re: CDN Overload? It appears all complaints are from SP doing wireless. I am going to go with a yes and put forth a these that these guys have a common factor somewhere. It could be equipment from a some popular vendor of wireless or maybe some common method to throttle that is popular in the wireless community. I note that while we have slow links we have no throttling or bandwidth management going on except for the buffering that happens in the DSLAM. Also there is no way to cheat. If you send 4 mbps to a 2 mbps DSL it will drop half of the traffic and TCP will not survive that. The CDN would have an effective transfer rate approaching zero for that customer. That seems to be a rather bad business proposal seen from the view if the CDN so they would not do that. The other customers will be unaffected as the DSLAM itself has plenty of capacity. Regards Baldur Den 21. sep. 2016 14.36 skrev "Josh Reynolds" : > With so many geographically diverse complaints on many hardware routing > and switching platforms, I'm going to go with a "no". > > On Sep 21, 2016 4:04 AM, "Baldur Norddahl" > wrote: > >> How come we have never seen this problem? We have a ton of DSL and many >> of those are slow, but no customer complaints about overloaded lines from >> CDN networks. >> >> Could it be that the way you throttle the bandwidth is defect? It is easy >> to blame the other guy but could it be that you are doing it wrong? >> >> Regards, >> >> Badur >> >> From beecher at beecher.cc Wed Sep 21 15:13:06 2016 From: beecher at beecher.cc (Tom Beecher) Date: Wed, 21 Sep 2016 11:13:06 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> References: <20160916131246.GP29651@dilbert.slimey.org> <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net> Message-ID: I have a hard time accepting that service providers should re-engineer their networks because other companies cannot properly engineer their abuse tooling. On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson wrote: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due to > them being hacked so many times getting them to actually communicate is > almost impossible. My .02 is just get the gamers a true public if at all > possible. > > Justin Wilson > j2sw at mtin.net > > --- > http://www.mtin.net Owner/CEO > xISP Solutions- Consulting ? Data Centers - Bandwidth > > http://www.midwest-ix.com COO/Chairman > Internet Exchange - Peering - Distributed Fabric > > > On Sep 20, 2016, at 8:24 AM, Danijel Starman > wrote: > > > > Something similar happened to a local FantasyConon I was helping set up, > we > > had only two PS4 machines there and accounts provided by Blizzard for > > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > > about 8h. There was no other traffic other then those two accounts > playing > > Overwatch so my guess is that they have some too aggressive checks. I've > > managed to convince our ISP there to change the outside IP of the link so > > we got them working the next day but it happened again in 8h. > > > > -- > > *blap* > > > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart > wrote: > > > >> All, > >> > >> We operate an access network with several hundred thousand users. > >> Increasingly > >> we're putting the users behind CGNAT in order to continue to give them > an > >> IPv4 > >> service (we're all dual-stack, so they all get public IPv6 too). Due to > the > >> demographic of our users, many of them are gamers. > >> > >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some > >> of our > >> CGNAT outside addresses, because they claim to have received anomalous, > or > >> 'attack' traffic from that IP. This obviously causes problems for the > other > >> legitimate users who end up behind the same public IPv4 address. > >> > >> Despite numerous attempts to engage with PSN, they are unwilling to > give us > >> any additional information which would allow us to identify the 'rogue' > >> users > >> on our network, or to identify the 'unwanted' traffic so that we could > >> either > >> block it, or use it to identify the rogue users ourselves. > >> > >> Has anyone else come up against the problem, and/or have any > suggestions on > >> how best to resolve it? > >> > >> Many thanks in advance, > >> > >> Simon > >> > >> > > > > From ainglis at agoc.com Mon Sep 19 18:03:01 2016 From: ainglis at agoc.com (Inglis, Adam) Date: Mon, 19 Sep 2016 18:03:01 +0000 Subject: Ebay operations contact please Message-ID: <013E121BB2474C489C9000DFA2239C3CBE9B357E@agcmbx03.AGC01.com> Can someone from ebay NOC (or similar) contact me off list for an issue with search results from a specific CIDR of our network that the end user swears doesn't happen elsewhere? Thanks. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender or send back to returns at agoc.com and delete the material from any computer. From jeff+nanog at waddellsolutions.com Tue Sep 20 21:25:31 2016 From: jeff+nanog at waddellsolutions.com (Jeff Waddell) Date: Tue, 20 Sep 2016 17:25:31 -0400 Subject: Level 3 GBLX / Legacy Twt IPv6 Issue Message-ID: I am having an issue where since the the old twt convergence by Level 3 to the GBLX network/AS3549 our v6 prefix isn't being announced outside of Level 3. I have had a ticket open for a week and had one engineer who seemed to be on to fixing it (regarding an export policy) but then that went silent and now I am getting no-where with support (other than promised call backs) Can someone from Level 3 reach out off list and help me out? Thanks Jeff jeff at waddellsolutions.com From jeff.jjones at gmail.com Mon Sep 19 17:19:43 2016 From: jeff.jjones at gmail.com (Jeff Jones) Date: Mon, 19 Sep 2016 13:19:43 -0400 Subject: Domain renawals Message-ID: Hello All, Sorry if this is low level. But are people sick of registrars jacking up prices? Who is the cheapest and most reliable? I have been using whois.com, networksolutions.com and am looking for input on who is cheap, secure, reliable registrar. Thanks for your input. ~Jeff From list-nanog at beufa.net Mon Sep 19 21:36:01 2016 From: list-nanog at beufa.net (Fabien V.) Date: Mon, 19 Sep 2016 21:36:01 +0000 Subject: ATT contact about BGP leak Message-ID: <83aa8ee2584b0d0a2e243247d99cbc19@mail.beufa.net> Hi the list =) Someone from AT&T on the list ? We have a potential BGP leak on a recently transfered /16 Sent email to noc @ but no reply since 6 hours. Perhaps someone can contact me directly ? fabien AT beufa DOT net Thanks ----- Fabien VINCENT ------------------------------------------------------------------- Twitter : @beufanet Website : https://beufa.net From sakamura at gmail.com Wed Sep 21 16:17:11 2016 From: sakamura at gmail.com (Ishmael Rufus) Date: Wed, 21 Sep 2016 11:17:11 -0500 Subject: Domain renawals In-Reply-To: References: Message-ID: $9.88 for commercial domains seems under the average from what I've seen from other registrars On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones wrote: > Hello All, > > Sorry if this is low level. But are people sick of registrars jacking up > prices? Who is the cheapest and most reliable? I have been using whois.com > , > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. > > ~Jeff > From Valdis.Kletnieks at vt.edu Wed Sep 21 17:16:50 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Wed, 21 Sep 2016 13:16:50 -0400 Subject: Domain renawals In-Reply-To: References: Message-ID: <10909.1474478210@turing-police.cc.vt.edu> On Mon, 19 Sep 2016 13:19:43 -0400, Jeff Jones said: > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. cheap, secure, reliable - pick any two. (The driver here is "cheap" - the other two criteria can be almost anything, but to do them well will probably not be cheap. Alternately, you can have three criteria that are all done excellently - but that level of service won't come cheap) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From jim at reptiles.org Wed Sep 21 17:18:10 2016 From: jim at reptiles.org (Jim Mercer) Date: Wed, 21 Sep 2016 13:18:10 -0400 Subject: Domain renawals In-Reply-To: References: Message-ID: <20160921171810.GA96327@reptiles.org> cheap, secure, reliable pick two. --jim On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones wrote: > Sorry if this is low level. But are people sick of registrars jacking up > prices? Who is the cheapest and most reliable? I have been using whois.com, > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. -- Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 Life should not be a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming "Wow! What a Ride!" -- Hunter S. Thompson From hvgeekwtrvl at gmail.com Wed Sep 21 18:43:50 2016 From: hvgeekwtrvl at gmail.com (james machado) Date: Wed, 21 Sep 2016 11:43:50 -0700 Subject: Domain renawals In-Reply-To: <20160921171810.GA96327@reptiles.org> References: <20160921171810.GA96327@reptiles.org> Message-ID: so who would you quantify as secure and reliable? who does not require additional "services" besides registration or spend all their time trying to upsell you? james On Wed, Sep 21, 2016 at 10:18 AM, Jim Mercer wrote: > > cheap, secure, reliable > > pick two. > > --jim > > On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones > wrote: > > Sorry if this is low level. But are people sick of registrars jacking up > > prices? Who is the cheapest and most reliable? I have been using > whois.com, > > networksolutions.com and am looking for input on who is cheap, secure, > > reliable registrar. Thanks for your input. > > -- > Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 > > Life should not be a journey to the grave with the intention of > arriving safely in a pretty and well preserved body, but rather > to skid in broadside in a cloud of smoke, thoroughly used up, > totally worn out, and loudly proclaiming "Wow! What a Ride!" > -- Hunter S. Thompson > From jim at reptiles.org Wed Sep 21 18:46:43 2016 From: jim at reptiles.org (Jim Mercer) Date: Wed, 21 Sep 2016 14:46:43 -0400 Subject: Domain renawals In-Reply-To: References: <20160921171810.GA96327@reptiles.org> Message-ID: <20160921184643.GA99310@reptiles.org> On Wed, Sep 21, 2016 at 11:43:50AM -0700, james machado wrote: > so who would you quantify as secure and reliable? who does not require > additional "services" besides registration or spend all their time trying > to upsell you? i'm good with easydns.com --jim > > james > > On Wed, Sep 21, 2016 at 10:18 AM, Jim Mercer wrote: > > > > > cheap, secure, reliable > > > > pick two. > > > > --jim > > > > On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones > > wrote: > > > Sorry if this is low level. But are people sick of registrars jacking up > > > prices? Who is the cheapest and most reliable? I have been using > > whois.com, > > > networksolutions.com and am looking for input on who is cheap, secure, > > > reliable registrar. Thanks for your input. > > > > -- > > Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 > > > > Life should not be a journey to the grave with the intention of > > arriving safely in a pretty and well preserved body, but rather > > to skid in broadside in a cloud of smoke, thoroughly used up, > > totally worn out, and loudly proclaiming "Wow! What a Ride!" > > -- Hunter S. Thompson > > -- Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 Life should not be a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming "Wow! What a Ride!" -- Hunter S. Thompson From mark at exonetric.com Wed Sep 21 18:49:31 2016 From: mark at exonetric.com (Mark Blackman) Date: Wed, 21 Sep 2016 19:49:31 +0100 Subject: Domain renawals In-Reply-To: References: <20160921171810.GA96327@reptiles.org> Message-ID: > On 21 Sep 2016, at 19:43, james machado wrote: > > so who would you quantify as secure and reliable? who does not require > additional "services" besides registration or spend all their time trying > to upsell you? > > james I have always liked https://www.gandi.net/ - Mark From rsk at gsp.org Wed Sep 21 19:09:31 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Wed, 21 Sep 2016 15:09:31 -0400 Subject: Domain renawals In-Reply-To: References: Message-ID: <20160921190931.GA23276@gsp.org> On Mon, Sep 19, 2016 at 01:19:43PM -0400, Jeff Jones wrote: Who is the cheapest and most reliable? I've had an excellent experience with NearlyFreeSpeech: https://www.nearlyfreespeech.net/ They have a high level of technical clue, don't try to upsell me things I don't need or want (although they do offer services), and they've been very efficient/precise about handling support requests. ---rsk From holbor at sonss.net Wed Sep 21 20:44:44 2016 From: holbor at sonss.net (Richard Holbo) Date: Wed, 21 Sep 2016 13:44:44 -0700 Subject: Domain renawals In-Reply-To: References: Message-ID: FWIW, as I'm in the middle of this right now. It would appear that many of the less expensive registrars no longer support glue records in any meaningful way. They all expect you to host DNS with them. So might want to check on that before buying the cheapest and hosting your own DNS. /rh On Mon, Sep 19, 2016 at 10:19 AM, Jeff Jones wrote: > Hello All, > > Sorry if this is low level. But are people sick of registrars jacking up > prices? Who is the cheapest and most reliable? I have been using whois.com > , > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. > > ~Jeff > From johnl at iecc.com Wed Sep 21 20:52:29 2016 From: johnl at iecc.com (John Levine) Date: 21 Sep 2016 20:52:29 -0000 Subject: Domain renawals In-Reply-To: Message-ID: <20160921205229.90252.qmail@ary.lan> In article you write: >FWIW, as I'm in the middle of this right now. It would appear that many of >the less expensive registrars no longer support glue records in any >meaningful way. They all expect you to host DNS with them. So might want >to check on that before buying the cheapest and hosting your own DNS. I resell Tucows, and glue records definitely work. You have to specifically export the ones you want, but when you do, it works. R's, John From surfer at mauigateway.com Wed Sep 21 21:10:43 2016 From: surfer at mauigateway.com (Scott Weeks) Date: Wed, 21 Sep 2016 14:10:43 -0700 Subject: CDN Overload? Message-ID: <20160921141043.760243D8@m0087798.ppops.net> :: I have made this into a Google Form to make it easier to :: track compared to randomly formatted responses on multiple :: mailing lists, Facebook Groups, etc. Yeah, because... but I don't do email like that why is it hard to read? it's really hard to read email this way. because it's out of order umm, ok. I fixed it for you scott From marka at isc.org Wed Sep 21 21:35:37 2016 From: marka at isc.org (Mark Andrews) Date: Thu, 22 Sep 2016 07:35:37 +1000 Subject: Domain renawals In-Reply-To: Your message of "Mon, 19 Sep 2016 13:19:43 -0400." References: Message-ID: <20160921213537.E1E7754C6035@rock.dv.isc.org> In message , Jeff Jones writes: > Hello All, > > Sorry if this is low level. But are people sick of registrars jacking up > prices? Who is the cheapest and most reliable? I have been using whois.com, > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. > > ~Jeff Remember to check that their DNS servers are RFC compliant. You can check EDNS compliance at https://ednscomp.isc.org/ednscomp Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From math at sizone.org Wed Sep 21 22:00:55 2016 From: math at sizone.org (Ken Chase) Date: Wed, 21 Sep 2016 18:00:55 -0400 Subject: Domain renawals In-Reply-To: <20160921184643.GA99310@reptiles.org> References: <20160921171810.GA96327@reptiles.org> <20160921184643.GA99310@reptiles.org> Message-ID: <20160921220055.GQ15891@sizone.org> EasyDNS has gone beyond the normal registrar dilligence and has resisted bogus takedowns and other things, where many would just bend over backwards. They can do this a bit more easily by being in Canada as well: https://www.techdirt.com/articles/20160606/10541834640/riaa-demands-takedown-thepiratebayorg-easydns-refuses-over-lack-due-process.shtml https://www.techdirt.com/articles/20150107/17585829627/easydns-sued-refusing-to-take-down-website-without-court-order-then-hit-again-writing-about-lawsuit.shtml https://www.techdirt.com/articles/20131127/02062025385/easydns-continues-to-fight-bogus-website-seizures-city-london-police-after-verisign-issues-no-decision.shtml https://www.techdirt.com/articles/20150623/17321931439/icanns-war-whois-privacy.shtml Mark Jeftovic, owner is a great guy who's one of the old school netheads (cut my teeth with him as co admins under an ISP owned by Osama Arafat who went on to found Q9). Recommended. /kc On Wed, Sep 21, 2016 at 02:46:43PM -0400, Jim Mercer said: >On Wed, Sep 21, 2016 at 11:43:50AM -0700, james machado wrote: >> so who would you quantify as secure and reliable? who does not require >> additional "services" besides registration or spend all their time trying >> to upsell you? > >i'm good with easydns.com > >--jim > >> >> james >> >> On Wed, Sep 21, 2016 at 10:18 AM, Jim Mercer wrote: >> >> > >> > cheap, secure, reliable >> > >> > pick two. >> > >> > --jim >> > >> > On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones >> > wrote: >> > > Sorry if this is low level. But are people sick of registrars jacking up >> > > prices? Who is the cheapest and most reliable? I have been using >> > whois.com, >> > > networksolutions.com and am looking for input on who is cheap, secure, >> > > reliable registrar. Thanks for your input. >> > >> > -- >> > Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 >> > >> > Life should not be a journey to the grave with the intention of >> > arriving safely in a pretty and well preserved body, but rather >> > to skid in broadside in a cloud of smoke, thoroughly used up, >> > totally worn out, and loudly proclaiming "Wow! What a Ride!" >> > -- Hunter S. Thompson >> > > >-- >Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 > >Life should not be a journey to the grave with the intention of >arriving safely in a pretty and well preserved body, but rather >to skid in broadside in a cloud of smoke, thoroughly used up, >totally worn out, and loudly proclaiming "Wow! What a Ride!" > -- Hunter S. Thompson -- Ken Chase - math at sizone.org Toronto Canada From chk at pobox.com Wed Sep 21 22:24:28 2016 From: chk at pobox.com (Harald Koch) Date: Wed, 21 Sep 2016 18:24:28 -0400 Subject: Domain renawals In-Reply-To: <20160921220055.GQ15891@sizone.org> References: <20160921171810.GA96327@reptiles.org> <20160921184643.GA99310@reptiles.org> <20160921220055.GQ15891@sizone.org> Message-ID: There are still many registrars that don't support DNSSEC (possibly only for a subset of TLDs), and/or have an unusable or cumbersome interface for adding DNSSEC glue. Just another thing to watch out for... From nanog at ics-il.net Thu Sep 22 00:30:52 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 21 Sep 2016 19:30:52 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> Message-ID: <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" To: "NANOG" Sent: Wednesday, September 21, 2016 9:08:55 AM Subject: Re: CDN Overload? https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" To: "NANOG" Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From beckman at angryox.com Thu Sep 22 01:14:59 2016 From: beckman at angryox.com (Peter Beckman) Date: Wed, 21 Sep 2016 21:14:59 -0400 Subject: Domain renawals In-Reply-To: References: Message-ID: I use DNS Made Easy for all of my DNS hosting, which I'm happy to recommend. For domain registration I found that joining the GoDaddy Domain Club ( $120/year or less if you pay ahead for multiple years [1] ) is a good deal for the quantity of domains I own (56 and counting). It's kind of like Sam's Club -- you pay a membership fee for lower bulk pricing. Additionally they handle nearly every TLD, like .us, .name and co.uk. NearlyFreeSpeech.net looks to have pricing that is close to that of the Domain Club, may have to check them out. The Domain Club cost of $120 divided by 56 domains is about $2.15 per Domain, so NearlyFree wins handily. I'd like to learn more about the WHO behind NFSN, as well as how and when they offer support. TLD NearlyFree GoDaddy Domain Club [Adjusted] com $9.34 > $8.29 [$10.44] org $11.39 < $11.99 [$14.14] net $10.54 > $8.99 [$11.14] info $10.69 > $9.99 [$12.14] name $8.99 < $9.99 [$12.14] biz $11.19 < $11.99 [$14.14] In the 10-15 years of using GoDaddy, despite my disagreement with some of their marketing and public business positions, my domains don't get stolen, they haven't shut anything down, I haven't lost a domain name, and their support is decent when I need it (and it is 24/7 phone / email / chat). [1] https://www.godaddy.com/domains/discount-domains.aspx Beckman On Mon, 19 Sep 2016, Jeff Jones wrote: > Hello All, > > Sorry if this is low level. But are people sick of registrars jacking up > prices? Who is the cheapest and most reliable? I have been using whois.com, > networksolutions.com and am looking for input on who is cheap, secure, > reliable registrar. Thanks for your input. > > ~Jeff > --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- From hannigan at gmail.com Thu Sep 22 01:19:35 2016 From: hannigan at gmail.com (Martin Hannigan) Date: Wed, 21 Sep 2016 15:19:35 -1000 Subject: CDN Overload? In-Reply-To: <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> Message-ID: Mike, I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. Jared alluded to fasttcp a few emails ago. Astute man. Best, Martin Hannigan AS 20940 // AS 32787 > On Sep 21, 2016, at 14:30, Mike Hammett wrote: > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing > > I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" > To: "NANOG" > Sent: Wednesday, September 21, 2016 9:08:55 AM > Subject: Re: CDN Overload? > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" > To: "NANOG" > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > > From nanog at ics-il.net Thu Sep 22 01:29:49 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 21 Sep 2016 20:29:49 -0500 (CDT) Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> Message-ID: <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> Thanks Marty. I have only experienced this on my network once and it was directly with Microsoft, so I haven't done much until a couple days ago when I started this campaign. I don't know if anyone else has brought this to anyone's attention. I just sent an e-mail to Owen when I saw yours. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Martin Hannigan" To: "Mike Hammett" Cc: "NANOG" Sent: Wednesday, September 21, 2016 8:19:35 PM Subject: Re: CDN Overload? Mike, I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. Jared alluded to fasttcp a few emails ago. Astute man. Best, Martin Hannigan AS 20940 // AS 32787 On Sep 21, 2016, at 14:30, Mike Hammett < nanog at ics-il.net > wrote: https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" < nanog at ics-il.net > To: "NANOG" < nanog at nanog.org > Sent: Wednesday, September 21, 2016 9:08:55 AM Subject: Re: CDN Overload? https://goo.gl/forms/LvgFRsMdNdI8E9HF3 I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" < nanog at ics-il.net > To: "NANOG" < nanog at nanog.org > Sent: Monday, September 19, 2016 12:34:48 PM Subject: CDN Overload? I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From johnl at iecc.com Thu Sep 22 01:35:55 2016 From: johnl at iecc.com (John Levine) Date: 22 Sep 2016 01:35:55 -0000 Subject: Domain renawals In-Reply-To: Message-ID: <20160922013555.90778.qmail@ary.lan> >For domain registration I found that joining the GoDaddy Domain Club >( $120/year or less if you pay ahead for multiple years [1] ) ... There's a lot of registrars with prepay discounts. Gandi's domains are cheaper if you prepay $600, a lot cheaper if you prepay $2000. R's, John From hannigan at gmail.com Thu Sep 22 01:40:58 2016 From: hannigan at gmail.com (Martin Hannigan) Date: Wed, 21 Sep 2016 15:40:58 -1000 Subject: CDN Overload? In-Reply-To: <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> Message-ID: No problem. If you can drop a pcap file somewhere we can reach (and drop me an email where) that was created during the event that'd be great. Thanks again, and great use of the list. Best, Martin Hannigan AS 20940 // AS 32787 > On Sep 21, 2016, at 15:29, Mike Hammett wrote: > > Thanks Marty. I have only experienced this on my network once and it was directly with Microsoft, so I haven't done much until a couple days ago when I started this campaign. I don't know if anyone else has brought this to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > From: "Martin Hannigan" > To: "Mike Hammett" > Cc: "NANOG" > Sent: Wednesday, September 21, 2016 8:19:35 PM > Subject: Re: CDN Overload? > > > Mike, > > I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. > > Jared alluded to fasttcp a few emails ago. Astute man. > > Best, > > Martin Hannigan > AS 20940 // AS 32787 > > > > On Sep 21, 2016, at 14:30, Mike Hammett wrote: > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing > > I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" > To: "NANOG" > Sent: Wednesday, September 21, 2016 9:08:55 AM > Subject: Re: CDN Overload? > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" > To: "NANOG" > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > > > From justin at cloudflare.com Thu Sep 22 02:10:35 2016 From: justin at cloudflare.com (Justin Paine) Date: Wed, 21 Sep 2016 19:10:35 -0700 Subject: Domain renawals In-Reply-To: <20160922013555.90778.qmail@ary.lan> References: <20160922013555.90778.qmail@ary.lan> Message-ID: I've had quite good luck with: Gandi, Hover, 101domains, and Google Domains -- depending on which cc/TLDs you're looking for. ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Wed, Sep 21, 2016 at 6:35 PM, John Levine wrote: >>For domain registration I found that joining the GoDaddy Domain Club >>( $120/year or less if you pay ahead for multiple years [1] ) ... > > There's a lot of registrars with prepay discounts. Gandi's domains > are cheaper if you prepay $600, a lot cheaper if you prepay $2000. > > R's, > John From mysidia at gmail.com Thu Sep 22 03:10:22 2016 From: mysidia at gmail.com (Jimmy Hess) Date: Wed, 21 Sep 2016 22:10:22 -0500 Subject: Domain renawals In-Reply-To: <20160922013555.90778.qmail@ary.lan> References: <20160922013555.90778.qmail@ary.lan> Message-ID: On Wed, Sep 21, 2016 at 8:35 PM, John Levine wrote: >>For domain registration I found that joining the GoDaddy Domain Club >>( $120/year or less if you pay ahead for multiple years [1] ) ... > There's a lot of registrars with prepay discounts. Gandi's domains > are cheaper if you prepay $600, a lot cheaper if you prepay $2000. Prepayment makes no sense, unless you are planning on maintaining more than 10 domains, which warrants much more due dilligence than if registering one or two domains. Also, if you're maintaining one or two domains, then it is sensible to pay more for a registrar that provides better support, or a more intuitive web interface. For maintaining a larger number of domains: perhaps more powerful management tools are more useful, and possibly the ease-of-use is a lower priority. Therefore, it depends on what you are doing with domains. I know of registrars that are $8.99 per Year and $8.39 per Year for a .COM, with no prepayment necessary, for those rates, and small discounts for prepay. * They say "cheap, secure, reliable, pick two" But that's not really how it is. it's really more like "Inexpensive, Good support, Feature-complete", pick two. Because no registrar is "secure" totally; phishing is conceivable with any registrar. That includes ne'er do wells pre-texting you and tricking registrar support personnel to change your e-mail address plus password and give it to a cracker. You can't give up reliability to get security, so the original 3 don't work. Every registrar known to offer advanced security mitigations charges a boatload, or part of a boatload to add them. If you want security, then the closest you get is what's called a Registry lock with' a telephone-based confirmation of domain changes, And two-factor login to the website. Last I check, getting the registry lock service is Only available on certain TLDs, and adds between $500 and $1000 Per domain name to the cost. Also, there is a bit of inconvenience, since you are setting a lock which your domain registrar is unable to override on their own, so routine maintenance such as updating DNS servers or renewing becomes a potentially drawn-out process..... Various registrars offer Two-Factor website login and 'Max Lock' features of their own, providing their own confirmation, and just a Client/Registrar-Lock on the domain, But again...... you can't see the registrar's IT systems, so blindly assuming they are secure would be silly. Certainly price can't tell you that. None of the registrars are going to be totally secure. It's just a question of.... How long have they been around, how much business does the registrar do, and how many times have they been hacked and the hack was bad enough that the internet community discovered it? -- -JH From lou at metron.com Thu Sep 22 03:28:42 2016 From: lou at metron.com (Lou Katz) Date: Wed, 21 Sep 2016 20:28:42 -0700 Subject: Domain renawals In-Reply-To: <20160921205229.90252.qmail@ary.lan> References: <20160921205229.90252.qmail@ary.lan> Message-ID: <20160922032842.GA57215@metron.com> On Wed, Sep 21, 2016 at 08:52:29PM -0000, John Levine wrote: > In article you write: > >FWIW, as I'm in the middle of this right now. It would appear that many of > >the less expensive registrars no longer support glue records in any > >meaningful way. They all expect you to host DNS with them. So might want > >to check on that before buying the cheapest and hosting your own DNS. > > I resell Tucows, and glue records definitely work. You have to > specifically export the ones you want, but when you do, it works. Me too. > > R's, > John -- -=[L]=- Composed on an ASR33 If you want to go somewhere, goto is the best way to get there. From beckman at angryox.com Thu Sep 22 03:46:07 2016 From: beckman at angryox.com (Peter Beckman) Date: Wed, 21 Sep 2016 23:46:07 -0400 Subject: Domain renawals In-Reply-To: <20160922013555.90778.qmail@ary.lan> References: <20160922013555.90778.qmail@ary.lan> Message-ID: On Wed, 22 Sep 2016, John Levine wrote: >> For domain registration I found that joining the GoDaddy Domain Club >> ( $120/year or less if you pay ahead for multiple years [1] ) ... > > There's a lot of registrars with prepay discounts. Gandi's domains > are cheaper if you prepay $600, a lot cheaper if you prepay $2000. I see the discount, and $600 prepay IS cheaper than Gandi rates with NO prepay. But the other companies are still less expensive even with the Gandi prepay. TLD NearlyFree GoDaddy DDC Gandi B Rates ($600) com $9.34 $10.44 $14.50 org $11.39 $14.14 $16.20 net $10.54 $11.14 $17.00 info $10.69 $12.14 $15.55 name $8.99 $12.14 $14.60 biz $11.19 $14.14 $16.28 Now if you get to $12,000 prepay, you get E Rates, where .com is $8.80 and .net is $11.00. Lower than most, but NearlyFree is still very competitive and even beats Gandi on a few TLDs at E Rates. I'm sure there are more benefits to Gandi over others than just price. I agree with the other poster that other dimensions are also important and valuable: support quality, security, policies, UI, ease of use, communication. Beckman NOTE: All rates quoted are RENEWAL rates, not transfer or new, as of 9/21/16. GoDaddy DDC rates are discounted and adjusted for 56 domains for the DDC fee of $120 per year. More domains == lower prices. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- From outsider at scarynet.org Thu Sep 22 08:35:01 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Thu, 22 Sep 2016 10:35:01 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: <3010begyh0g1jvrtfhr2bkar.1474533301508@email.android.com> Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen..... Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Mark Andrews Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson Cc: NANOG Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP.? This results in them banning folks.? Due > to them being hacked so many times getting them to actually communicate > is almost impossible.? My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j2sw at mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6.? 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742???????????????? INTERNET: marka at isc.org From outsider at scarynet.org Thu Sep 22 08:42:20 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Thu, 22 Sep 2016 10:42:20 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: So you ignore/don't deal with the abuse coz it's shipped in a format you refuse to handle? And you don't even bother telling the reporter you would like it in a per ip format? Or make attempts to make it work the way they report it (split out the ip's and modify the to be forwarded mail to only contain the ip's belonging to that customer)???? Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Baldur Norddahl Datum: 21-09-16 10:37 (GMT+01:00) Aan: nanog at nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses Hi We have the opposite problem with PSN: Sometimes they will send abuse reports with several of our IP addresses listed. The problem with that is that we can not give data about one customer to another customer. By listing multiple IP addresses we are prevented from forwarding the email to the customer. Which means we may ignore it instead. Regards, Baldur From outsider at scarynet.org Thu Sep 22 08:48:39 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Thu, 22 Sep 2016 10:48:39 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: As long as their is no international accepted standard as to how to report abuse and everyone cooking up his/her own methods.. I think you have either the choice of adapting and thus be able to deal with the abuse. Or be lazy and stubborn, ignore it, wait for the bad reputation to say hi to your company and face the effects it might cause. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Tom Beecher Datum: 21-09-16 17:13 (GMT+01:00) Aan: Justin Wilson Cc: NANOG Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses I have a hard time accepting that service providers should re-engineer their networks because other companies cannot properly engineer their abuse tooling. On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson wrote: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP.? This results in them banning folks.? Due to > them being hacked so many times getting them to actually communicate is > almost impossible.? My .02 is just get the gamers a true public if at all > possible. > > Justin Wilson > j2sw at mtin.net > > --- > http://www.mtin.net Owner/CEO > xISP Solutions- Consulting ? Data Centers - Bandwidth > > http://www.midwest-ix.com? COO/Chairman > Internet Exchange - Peering - Distributed Fabric > > > On Sep 20, 2016, at 8:24 AM, Danijel Starman > wrote: > > > > Something similar happened to a local FantasyConon I was helping set up, > we > > had only two PS4 machines there and accounts provided by Blizzard for > > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in > > about 8h. There was no other traffic other then those two accounts > playing > > Overwatch so my guess is that they have some too aggressive checks. I've > > managed to convince our ISP there to change the outside IP of the link so > > we got them working the next day but it happened again in 8h. > > > > -- > > *blap* > > > > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart > wrote: > > > >> All, > >> > >> We operate an access network with several hundred thousand users. > >> Increasingly > >> we're putting the users behind CGNAT in order to continue to give them > an > >> IPv4 > >> service (we're all dual-stack, so they all get public IPv6 too). Due to > the > >> demographic of our users, many of them are gamers. > >> > >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some > >> of our > >> CGNAT outside addresses, because they claim to have received anomalous, > or > >> 'attack' traffic from that IP. This obviously causes problems for the > other > >> legitimate users who end up behind the same public IPv4 address. > >> > >> Despite numerous attempts to engage with PSN, they are unwilling to > give us > >> any additional information which would allow us to identify the 'rogue' > >> users > >> on our network, or to identify the 'unwanted' traffic so that we could > >> either > >> block it, or use it to identify the rogue users ourselves. > >> > >> Has anyone else come up against the problem, and/or have any > suggestions on > >> how best to resolve it? > >> > >> Many thanks in advance, > >> > >> Simon > >> > >> > > > > From nanog at ics-il.net Thu Sep 22 11:23:49 2016 From: nanog at ics-il.net (Mike Hammett) Date: Thu, 22 Sep 2016 06:23:49 -0500 (CDT) Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <3010begyh0g1jvrtfhr2bkar.1474533301508@email.android.com> References: <3010begyh0g1jvrtfhr2bkar.1474533301508@email.android.com> Message-ID: <1817394378.16169.1474543429219.JavaMail.mhammett@ThunderFuck> If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care. ;-) They know enough to hate NAT. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Alexander Maassen" Cc: "NANOG" Sent: Thursday, September 22, 2016 3:35:01 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen..... Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Mark Andrews Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson Cc: NANOG Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP. This results in them banning folks. Due > to them being hacked so many times getting them to actually communicate > is almost impossible. My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j2sw at mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6. 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From baldur.norddahl at gmail.com Thu Sep 22 12:10:07 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Thu, 22 Sep 2016 14:10:07 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: Message-ID: On 22 September 2016 at 10:42, Alexander Maassen wrote: > So you ignore/don't deal with the abuse coz it's shipped in a format you > refuse to handle? > > And you don't even bother telling the reporter you would like it in a per > ip format? Or make attempts to make it work the way they report it (split > out the ip's and modify the to be forwarded mail to only contain the ip's > belonging to that customer)???? > You will have to remember that these are automated mails from the reporter. If I write them back it goes into their bit bucket, because they do not really care enough to bother replying. I am betting they are sending out thousands mails each day and they can not handle manually replying to all of that. In the same way we receive a large amount of automated mail so we have to be able to handle it automatically. Send me something sane and I will make a script that forwards it. Send me something unusable and I wont - but I will not do manual handling of your automated mail. All I am trying to do here is tell people that send abuse mails not to combine multiple abuse complaints in one mail, because that makes it harder for everybody and makes it more likely that your mail will be dropped as too much work. Double so if your abuse mails is from an automated system, because I will try to match your automated system with my own. However it is much harder to make a system that can edit your complaint and duplicate it to several recipients, than it is to run a simple filter that just forwards the mail as is. As to PSN they will usually send multiple mails if the abuse is ongoing. At some point they will send a mail with just one IP and that one gets forwarded. So we are dropping some of the mails, but the users eventually get notified anyway. It is not ideal but it works. Regards, Baldur From outsider at scarynet.org Thu Sep 22 12:31:12 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Thu, 22 Sep 2016 14:31:12 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: Maybe its time then for a global accepted, unified way to send/report abuse?? That should solve most of the issues and end points would be able to deal with it in a common way and only would need to think about how to integrate it in their crm's etc. We are all using the same medium, but attempt to communicate issues using several methods.? Perhaps iana can use those (m/b)illions they got from selling tld's and cook something up. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Baldur Norddahl Datum: 22-09-16 14:10 (GMT+01:00) Aan: nanog at nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses On 22 September 2016 at 10:42, Alexander Maassen wrote: > So you ignore/don't deal with the abuse coz it's shipped in a format you > refuse to handle? > > And you don't even bother telling the reporter you would like it in a per > ip format? Or make attempts to make it work the way they report it (split > out the ip's and modify the to be forwarded mail to only contain the ip's > belonging to that customer)???? > You will have to remember that these are automated mails from the reporter. If I write them back it goes into their bit bucket, because they do not really care enough to bother replying. I am betting they are sending out thousands mails each day and they can not handle manually replying to all of that. In the same way we receive a large amount of automated mail so we have to be able to handle it automatically. Send me something sane and I will make a script that forwards it. Send me something unusable and I wont - but I will not do manual handling of your automated mail. All I am trying to do here is tell people that send abuse mails not to combine multiple abuse complaints in one mail, because that makes it harder for everybody and makes it more likely that your mail will be dropped as too much work. Double so if your abuse mails is from an automated system, because I will try to match your automated system with my own. However it is much harder to make a system that can edit your complaint and duplicate it to several recipients, than it is to run a simple filter that just forwards the mail as is. As to PSN they will usually send multiple mails if the abuse is ongoing. At some point they will send a mail with just one IP and that one gets forwarded. So we are dropping some of the mails, but the users eventually get notified anyway. It is not ideal but it works. Regards, Baldur From outsider at scarynet.org Thu Sep 22 12:45:04 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Thu, 22 Sep 2016 14:45:04 +0200 Subject: PlayStationNetwork blocking of CGNAT public addresses Message-ID: <5xj0xk01c55guyxgvo15sjvr.1474548304198@email.android.com> Ipv6 is there for 20+ years, cgnat is needed coz the net grows kinda exponentially due to stuff like IoT/mobiles/m2m, and isp's need to provide users with the ability to talk ipv4 simply because the other side refuses to deploy v6 abilities. Do the math if they really care. Also the servers itself hosting the gameserver probably already are dual stacked. But the gamecode itself misses the support. Then there is the issue of you as isp not being able or daring to show a fist and simply saying: screw you. Because you are risking to loose customers. And as long as the company's earn plenty of money using outdated code, they won't change it, coz that would imply spending money that won't flow into fancy buildings, fast cars and all that other useless luxury. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Mike Hammett Datum: 22-09-16 13:23 (GMT+01:00) Aan: Alexander Maassen Cc: NANOG Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care.? ;-) They know enough to hate NAT. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP From: "Alexander Maassen" Cc: "NANOG" Sent: Thursday, September 22, 2016 3:35:01 AM Subject: Re: PlayStationNetwork blocking of CGNAT public addresses Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides. If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. Hence again I'd be voting for an ipv6 only day, but that will never happen..... Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Mark Andrews Datum: 21-09-16 ?03:29 ?(GMT+01:00) Aan: Justin Wilson Cc: NANOG Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net>, Justin Wilson write s: > PSN is one reason I am not a fan of CGNAT. All they see are tons of > connections from the same IP.? This results in them banning folks.? Due > to them being hacked so many times getting them to actually communicate > is almost impossible.? My .02 is just get the gamers a true public if at > all possible. > > Justin Wilson > j2sw at mtin.net What we need is business tech reporters to continually report on these failures of content providers to deliver their services over IPv6.? 20 years lead time should be enough for any service. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742???????????????? INTERNET: marka at isc.org From cb.list6 at gmail.com Thu Sep 22 12:52:46 2016 From: cb.list6 at gmail.com (Ca By) Date: Thu, 22 Sep 2016 05:52:46 -0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <3010begyh0g1jvrtfhr2bkar.1474533301508@email.android.com> References: <3010begyh0g1jvrtfhr2bkar.1474533301508@email.android.com> Message-ID: On Thursday, September 22, 2016, Alexander Maassen wrote: > Both gamers and content providers do not care. The gamers as they only > care about the game itself and don't care about the technical mumbo jumbo. > And the makers coz they only care about making money by producing content > the gamers want. And you service providers are left with the headache of > attempts to please both sides. Very much agree > If this wasn't the case, then why after 20 years, ipv6 ain't rolled out. > Hence again I'd be voting for an ipv6 only day, but that will never > happen..... Disagree. IPv6 is meaningfully rolled out. Half or comcast and at&t subs are observably on ipv6 http://www.worldipv6launch.org/measurements/ And every (i think) iphone 7 ships with ipv6 default on from t-mobile, sprint, T , and VZ. Same can be said of samsung phones 2 years ago. Now, if abc isp and xyz gaming company don't deploy ipv6, they have nobody to blame but themselves. Many of us have moved on, but it is sad when you all need help tweeking your cgn or need help finding an IPv4 broker. I feel your pain. But don't say ipv6 is not deployed. It is deployed, and it carries more traffic than ipv4 http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ CB Kind regards, > Alexander Maassen > - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- > Peplink Certified Engineer > > -------- Oorspronkelijk bericht --------Van: Mark Andrews > Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson < > lists at mtin.net > Cc: NANOG > > Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses > > In message <09342130-874F-4FA4-B410-B7B66A75FA4D at mtin.net >, > Justin Wilson write > s: > > PSN is one reason I am not a fan of CGNAT. All they see are tons of > > connections from the same IP. This results in them banning folks. Due > > to them being hacked so many times getting them to actually communicate > > is almost impossible. My .02 is just get the gamers a true public if at > > all possible. > > > > Justin Wilson > > j2sw at mtin.net > > What we need is business tech reporters to continually report on > these failures of content providers to deliver their services over > IPv6. 20 years lead time should be enough for any service. > > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org > > From brak at gameservers.com Thu Sep 22 13:28:56 2016 From: brak at gameservers.com (Brian Rak) Date: Thu, 22 Sep 2016 09:28:56 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: Message-ID: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> On 9/22/2016 8:10 AM, Baldur Norddahl wrote: > On 22 September 2016 at 10:42, Alexander Maassen > wrote: > >> So you ignore/don't deal with the abuse coz it's shipped in a format you >> refuse to handle? >> >> And you don't even bother telling the reporter you would like it in a per >> ip format? Or make attempts to make it work the way they report it (split >> out the ip's and modify the to be forwarded mail to only contain the ip's >> belonging to that customer)???? >> > You will have to remember that these are automated mails from the reporter. > If I write them back it goes into their bit bucket, because they do not > really care enough to bother replying. I am betting they are sending out > thousands mails each day and they can not handle manually replying to all > of that. In the same way we receive a large amount of automated mail so we > have to be able to handle it automatically. Send me something sane and I > will make a script that forwards it. Send me something unusable and I wont > - but I will not do manual handling of your automated mail. > > All I am trying to do here is tell people that send abuse mails not to > combine multiple abuse complaints in one mail, because that makes it harder > for everybody and makes it more likely that your mail will be dropped as > too much work. Double so if your abuse mails is from an automated system, > because I will try to match your automated system with my own. However it > is much harder to make a system that can edit your complaint and duplicate > it to several recipients, than it is to run a simple filter that just > forwards the mail as is. > > As to PSN they will usually send multiple mails if the abuse is ongoing. At > some point they will send a mail with just one IP and that one gets > forwarded. So we are dropping some of the mails, but the users eventually > get notified anyway. It is not ideal but it works. > > Regards, > > Baldur We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > P.S. If you would prefer an individual email for each IP address on this list, please let us know. But, they didn't respond after we contacted them requesting it (and that line has since been removed). From ops.lists at gmail.com Thu Sep 22 13:34:39 2016 From: ops.lists at gmail.com (Suresh Ramasubramanian) Date: Thu, 22 Sep 2016 19:04:39 +0530 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> References: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> Message-ID: <42AF670D-A1A2-4495-A8FF-D4DC52BC9A5D@gmail.com> Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you? And how much time would processing say 50 individual emails take compared to 50 IPs in a single email? --srs > On 22-Sep-2016, at 6:58 PM, Brian Rak wrote: > > We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > > > P.S. If you would prefer an individual email for each IP address on this list, please let us know. > > But, they didn't respond after we contacted them requesting it (and that line has since been removed). From brak at gameservers.com Thu Sep 22 13:37:54 2016 From: brak at gameservers.com (Brian Rak) Date: Thu, 22 Sep 2016 09:37:54 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <42AF670D-A1A2-4495-A8FF-D4DC52BC9A5D@gmail.com> References: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> <42AF670D-A1A2-4495-A8FF-D4DC52BC9A5D@gmail.com> Message-ID: <88ca1bb5-9108-126b-6f84-3407ad688006@gameservers.com> Single IP per email: automated, zero time at all. Multiple IPs per email: manual process, minutes per IP. On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: > Considering that there are likely to be many such emails - just how > much time is it going to take your abuse desk staffer to just parse > out those IPs from whatever log that they send you? > > And how much time would processing say 50 individual emails take > compared to 50 IPs in a single email? > > --srs > > On 22-Sep-2016, at 6:58 PM, Brian Rak > wrote: > >> We've also started ignoring their abuse emails, for the same reason. >> Their abuse emails at one point contained the line: >> >> > P.S. If you would prefer an individual email for each IP address on this list, please let us know. >> >> But, they didn't respond after we contacted them requesting it (and >> that line has since been removed). From beecher at beecher.cc Thu Sep 22 14:05:07 2016 From: beecher at beecher.cc (Tom Beecher) Date: Thu, 22 Sep 2016 10:05:07 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <88ca1bb5-9108-126b-6f84-3407ad688006@gameservers.com> References: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> <42AF670D-A1A2-4495-A8FF-D4DC52BC9A5D@gmail.com> <88ca1bb5-9108-126b-6f84-3407ad688006@gameservers.com> Message-ID: The format of the abuse complaint doesn't mean anything if it still doesn't contain any relevant data to say what the abuse IS. (Or, even if it IS abuse at all.) On Thu, Sep 22, 2016 at 9:37 AM, Brian Rak wrote: > Single IP per email: automated, zero time at all. > > Multiple IPs per email: manual process, minutes per IP. > > > On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: > >> Considering that there are likely to be many such emails - just how much >> time is it going to take your abuse desk staffer to just parse out those >> IPs from whatever log that they send you? >> >> And how much time would processing say 50 individual emails take compared >> to 50 IPs in a single email? >> >> --srs >> >> On 22-Sep-2016, at 6:58 PM, Brian Rak > brak at gameservers.com>> wrote: >> >> We've also started ignoring their abuse emails, for the same reason. >>> Their abuse emails at one point contained the line: >>> >>> > P.S. If you would prefer an individual email for each IP address on >>> this list, please let us know. >>> >>> But, they didn't respond after we contacted them requesting it (and that >>> line has since been removed). >>> >> > From joly at punkcast.com Thu Sep 22 14:11:15 2016 From: joly at punkcast.com (Joly MacFie) Date: Thu, 22 Sep 2016 10:11:15 -0400 Subject: Livestream of NYNOG Meetup #2 today Message-ID: 10:45am EDT https://livestream.com/internetsociety/nynog2 -- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast -------------------------------------------------------------- - From Valdis.Kletnieks at vt.edu Thu Sep 22 14:23:44 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Thu, 22 Sep 2016 10:23:44 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: Message-ID: <73955.1474554224@turing-police.cc.vt.edu> On Thu, 22 Sep 2016 14:31:12 +0200, Alexander Maassen said: > Maybe its time then for a global accepted, unified way to send/report abuse? YOu mean ike these RFCs? (OK, so it's an XML schema. Just be glad it isn't ASN.1 :) 5070 The Incident Object Description Exchange Format. R. Danyliw, J. Meijer, Y. Demchenko. December 2007. (Format: TXT=171529 bytes) (Updated by RFC6685) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC5070) 6684 Guidelines and Template for Defining Extensions to the Incident Object Description Exchange Format (IODEF). B. Trammell. July 2012. (Format: TXT=23550 bytes) (Status: INFORMATIONAL) (DOI: 10.17487/RFC6684) 6685 Expert Review for Incident Object Description Exchange Format (IODEF) Extensions in IANA XML Registry. B. Trammell. July 2012. (Format: TXT=4363 bytes) (Updates RFC5070) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC6685) 7203 An Incident Object Description Exchange Format (IODEF) Extension for Structured Cybersecurity Information. T. Takahashi, K. Landfield, Y. Kadobayashi. April 2014. (Format: TXT=57694 bytes) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC7203) 7495 Enumeration Reference Format for the Incident Object Description Exchange Format (IODEF). A. Montville, D. Black. March 2015. (Format: TXT=19891 bytes) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC7495) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From hugo at slabnet.com Thu Sep 22 14:23:28 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Thu, 22 Sep 2016 07:23:28 -0700 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: Message-ID: <2F081138-DE7C-46DA-815D-CCC287A0A3D5@slabnet.com> http://x-arf.org/ ? -- Hugo Slabbert?????? | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E?? | also on Signal On September 22, 2016 5:31:12 AM PDT, Alexander Maassen wrote: >Maybe its time then for a global accepted, unified way to send/report >abuse?? >That should solve most of the issues and end points would be able to >deal with it in a common way and only would need to think about how to >integrate it in their crm's etc. >We are all using the same medium, but attempt to communicate issues >using several methods.? >Perhaps iana can use those (m/b)illions they got from selling tld's and >cook something up. > > > >Kind regards, >Alexander Maassen >- Technical Maintenance Engineer Parkstad Support BV- Maintainer >DroneBL- Peplink Certified Engineer > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 850 bytes Desc: not available URL: From ops.lists at gmail.com Thu Sep 22 14:26:18 2016 From: ops.lists at gmail.com (Suresh Ramasubramanian) Date: Thu, 22 Sep 2016 19:56:18 +0530 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: <6e01ae26-e887-10f2-1d8c-9a062a4aa072@gameservers.com> <42AF670D-A1A2-4495-A8FF-D4DC52BC9A5D@gmail.com> <88ca1bb5-9108-126b-6f84-3407ad688006@gameservers.com> Message-ID: Well yes ? if you have the automation, that is great. Of course the format of whatever log they send you matters too. I?ve had abuse complaints in a past life where the abuse report was a screenshot from a checkpoint firewall with ?Dear team, for your attention? in bright red in a large font. Personally I don?t trash abuse reports that are valid. --srs From: Tom Beecher Date: Thursday, 22 September 2016 at 7:35 PM To: Brian Rak Cc: Suresh Ramasubramanian , "nanog at nanog.org" Subject: Re: PlayStationNetwork blocking of CGNAT public addresses The format of the abuse complaint doesn't mean anything if it still doesn't contain any relevant data to say what the abuse IS. (Or, even if it IS abuse at all.) On Thu, Sep 22, 2016 at 9:37 AM, Brian Rak wrote: Single IP per email: automated, zero time at all. Multiple IPs per email: manual process, minutes per IP. On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote: Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you? And how much time would processing say 50 individual emails take compared to 50 IPs in a single email? --srs On 22-Sep-2016, at 6:58 PM, Brian Rak > wrote: We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line: > P.S. If you would prefer an individual email for each IP address on this list, please let us know. But, they didn't respond after we contacted them requesting it (and that line has since been removed). From dougb at dougbarton.us Thu Sep 22 14:37:21 2016 From: dougb at dougbarton.us (Doug Barton) Date: Thu, 22 Sep 2016 07:37:21 -0700 Subject: Domain renawals In-Reply-To: References: Message-ID: <187358fd-e334-2e50-7f03-b0fb47b8535d@dougbarton.us> On 09/21/2016 01:44 PM, Richard Holbo wrote: > FWIW, as I'm in the middle of this right now. It would appear that many of > the less expensive registrars no longer support glue records in any > meaningful way. They all expect you to host DNS with them. So might want > to check on that before buying the cheapest and hosting your own DNS. What do you think glue records are, and why do you think you need them? :) (Those are serious questions, btw) Doug From mysidia at gmail.com Thu Sep 22 16:15:49 2016 From: mysidia at gmail.com (Jimmy Hess) Date: Thu, 22 Sep 2016 11:15:49 -0500 Subject: Domain renawals In-Reply-To: <187358fd-e334-2e50-7f03-b0fb47b8535d@dougbarton.us> References: <187358fd-e334-2e50-7f03-b0fb47b8535d@dougbarton.us> Message-ID: On Thu, Sep 22, 2016 at 9:37 AM, Doug Barton wrote: > On 09/21/2016 01:44 PM, Richard Holbo wrote: >> FWIW, as I'm in the middle of this right now. It would appear that many of > What do you think glue records are, and why do you think you need them? :) > (Those are serious questions, btw) Glue records are also called "Host records", or Alternatively called: "Nameserver" records. These are A and AAAA records for your domain name which appear in the parent TLD zone, instead of the child zone. Host records also typically appear in WHOIS, for example: "$ whois ns5.yahoo.com" If you think your registrar does not support them, then you're probably having trouble with your registrar's user interface, and just don't have the right procedure, because the use of host records is quite essential and necessary for at least one domain to self-host DNS...... These records are non-authoritative and belong to the reply delegating nameservers for your domain to your servers, and you need to duplicate a copy of all your NS, A, AAAA records in your child zone, which must be identical to the parent's version of the records. For example, suppose your domain name is "Example.com" And you want your nameservers to be NS1.example.com, NS2.example.com, NS3.example.com. Because the nameservers exist in the same domain name which references them, the required DNS lookup graph is circular, and your DNS zone becomes an island! In order for clients to find your nameserver to figure out what NS1.example.com resolves to, it first needs to be able to find a nameserver for Example.com, which is NS1.example.com. This is what is circular without a Hint in the Additional section of the DNS reply from the parent nameserver. The glue record in the parent zone is used to tell the parent TLD server to include the IP address of your nameserver in the Additional Section of the DNS reply, so you can bootstrap DNS resolution for Example.com. > Doug -- -JH From johnl at iecc.com Thu Sep 22 16:30:21 2016 From: johnl at iecc.com (John Levine) Date: 22 Sep 2016 16:30:21 -0000 Subject: Domain renawals In-Reply-To: Message-ID: <20160922163021.93965.qmail@ary.lan> >In order for clients to find your nameserver to figure out what >NS1.example.com resolves to, >it first needs to be able to find a nameserver for Example.com, >which is NS1.example.com. > >This is what is circular without a Hint in the Additional section of >the DNS reply from the parent nameserver. That's true, but there's also cross-domain name servers. I have domains in .org with nameservers in .com. The org. registry won't publish the NS records if my registrar hasn't told the .com registry to push the name server records out to other TLDs. Those are two different kinds of glue, but in practice you need both. You could argue that .org doesn't need to publish cross-domain glue, and you would be right, but you still won't get your .org NS published if they don't have the glue. R's, John From matthew at corp.crocker.com Thu Sep 22 17:33:26 2016 From: matthew at corp.crocker.com (Matthew Crocker) Date: Thu, 22 Sep 2016 17:33:26 +0000 Subject: Lightower (ASN:46887) RTBH community info Message-ID: Hello, Does anyone know the RTBH community for Lightower? I?ve tried 46887:666 but that doesn?t work. I have a /32 I need to blackhole, Lightower is the last ISP and it doesn?t appear they support RTBH ? Thanks -Matt -- Matthew Crocker President ? Crocker Communications matthew at corp.crocker.com From holbor at sonss.net Thu Sep 22 18:44:25 2016 From: holbor at sonss.net (Richard Holbo) Date: Thu, 22 Sep 2016 11:44:25 -0700 Subject: Domain renawals In-Reply-To: References: <187358fd-e334-2e50-7f03-b0fb47b8535d@dougbarton.us> Message-ID: Since the circular notion of why we need glue records has already been addressed, I won't hit that here... I would agree with "you're probably having trouble with your registrar's user interface". In doing some work for a company that had a number of domains registered at 1and1.com, they (1and1) have a webpage about how to setup glue records, talks about it, but it does not work, and when you call their support, (google has many descriptions of the same issue)... they say that the only way it works is if you host your DNS with them, which kinda defeats the purpose. Whether this is just bad UI, bad support, or they just don't think it's necessary for most of their business ... does not really matter, effectively they are telling the customer who needs that to go somewhere else. In that process (going somewhere else) I've discovered that some registrars make it pretty easy, some ignore it completely. As there are probably fairly few of us than actually need this functionality I think a lot of less expensive registrars, just ignore it. Just throwing this out there in response to OP as something to watch out for because if you need it... Netsol, and Hover make it easy, Godaddy is not intuitive but doable FYI, IMHO. (DISCLAIMER not a complete list just my current limited experience, not meant to denigrate any other registrar that's not mentioned, please no flames). /rh On Thu, Sep 22, 2016 at 9:15 AM, Jimmy Hess wrote: > On Thu, Sep 22, 2016 at 9:37 AM, Doug Barton wrote: > > On 09/21/2016 01:44 PM, Richard Holbo wrote: > >> FWIW, as I'm in the middle of this right now. It would appear that many > of > > What do you think glue records are, and why do you think you need them? > :) > > (Those are serious questions, btw) > > Glue records are also called "Host records", or Alternatively > called: "Nameserver" records. > These are A and AAAA records for your domain name which appear in the > parent TLD zone, > instead of the child zone. > > Host records also typically appear in WHOIS, for example: "$ whois > ns5.yahoo.com" > > If you think your registrar does not support them, then you're > probably having trouble with > your registrar's user interface, and just don't have the right > procedure, because the use > of host records is quite essential and necessary for at least one > domain to self-host DNS...... > > > These records are non-authoritative and belong to the reply delegating > nameservers for > your domain to your servers, and you need to duplicate a copy of all > your NS, A, AAAA records in your > child zone, which must be identical to the parent's version of the > records. > > For example, suppose your domain name is "Example.com" > And you want your nameservers to be NS1.example.com, > NS2.example.com, NS3.example.com. > > Because the nameservers exist in the same domain name which references > them, > the required DNS lookup graph is circular, and your DNS zone becomes an > island! > > In order for clients to find your nameserver to figure out what > NS1.example.com resolves to, > it first needs to be able to find a nameserver for Example.com, > which is NS1.example.com. > > This is what is circular without a Hint in the Additional section of > the DNS reply from the parent nameserver. > > The glue record in the parent zone is used to tell the parent TLD > server to include the IP address of > your nameserver in the Additional Section of the DNS reply, so you > can bootstrap DNS resolution > for Example.com. > > > > > Doug > -- > -JH > From marka at isc.org Thu Sep 22 19:28:33 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 23 Sep 2016 05:28:33 +1000 Subject: Domain renawals In-Reply-To: Your message of "Thu, 22 Sep 2016 11:15:49 -0500." References: <187358fd-e334-2e50-7f03-b0fb47b8535d@dougbarton.us> Message-ID: <20160922192833.C456554CF317@rock.dv.isc.org> In message , Jimmy Hess writes: > On Thu, Sep 22, 2016 at 9:37 AM, Doug Barton wrote: > > On 09/21/2016 01:44 PM, Richard Holbo wrote: > >> FWIW, as I'm in the middle of this right now. It would appear that many of > > What do you think glue records are, and why do you think you need them? :) > > (Those are serious questions, btw) > > Glue records are also called "Host records", or Alternatively > called: "Nameserver" records. > These are A and AAAA records for your domain name which appear in the > parent TLD zone, > instead of the child zone. No. They are COPIES of records held in the parent zone to bootstrap access to the zone. They NEED to appear in BOTH places. If you only have them in the parent zone then you have a broken delegation which will fail intermittently for some resolvers when they learn that there are no records in the child zone. Named has code to detect this delegation error when it goes to load a zone. It blocks the load of the zone until you FIX the problem. It does this so that the error is made visible to the operator of the zone. > Host records also typically appear in WHOIS, for example: "$ whois > ns5.yahoo.com" > > If you think your registrar does not support them, then you're > probably having trouble with > your registrar's user interface, and just don't have the right > procedure, because the use > of host records is quite essential and necessary for at least one > domain to self-host DNS...... > > > These records are non-authoritative and belong to the reply delegating > nameservers for > your domain to your servers, and you need to duplicate a copy of all > your NS, A, AAAA records in your > child zone, which must be identical to the parent's version of the records. > > For example, suppose your domain name is "Example.com" > And you want your nameservers to be NS1.example.com, > NS2.example.com, NS3.example.com. > > Because the nameservers exist in the same domain name which references them, > the required DNS lookup graph is circular, and your DNS zone becomes an isla > nd! > > In order for clients to find your nameserver to figure out what > NS1.example.com resolves to, > it first needs to be able to find a nameserver for Example.com, > which is NS1.example.com. > > This is what is circular without a Hint in the Additional section of > the DNS reply from the parent nameserver. > > The glue record in the parent zone is used to tell the parent TLD > server to include the IP address of > your nameserver in the Additional Section of the DNS reply, so you > can bootstrap DNS resolution > for Example.com. > > > > > Doug > -- > -JH -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From mikeal.clark at gmail.com Thu Sep 22 19:32:52 2016 From: mikeal.clark at gmail.com (Mikeal Clark) Date: Thu, 22 Sep 2016 14:32:52 -0500 Subject: Domain renawals In-Reply-To: References: Message-ID: Yes I've had issues with providers dropping glue records! I have to email, and have a "advanced" support person fix them. The interface to do it myself has disappeared. Very frustrating. On Wed, Sep 21, 2016 at 3:44 PM, Richard Holbo wrote: > FWIW, as I'm in the middle of this right now. It would appear that many of > the less expensive registrars no longer support glue records in any > meaningful way. They all expect you to host DNS with them. So might want > to check on that before buying the cheapest and hosting your own DNS. > /rh > > On Mon, Sep 19, 2016 at 10:19 AM, Jeff Jones > wrote: > > > Hello All, > > > > Sorry if this is low level. But are people sick of registrars jacking up > > prices? Who is the cheapest and most reliable? I have been using > whois.com > > , > > networksolutions.com and am looking for input on who is cheap, secure, > > reliable registrar. Thanks for your input. > > > > ~Jeff > > > From bruce.curtis at ndsu.edu Thu Sep 22 21:28:17 2016 From: bruce.curtis at ndsu.edu (Bruce Curtis) Date: Thu, 22 Sep 2016 21:28:17 +0000 Subject: CDN Overload? In-Reply-To: <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> Message-ID: <35119344-8731-4C6F-A561-E2E7B43D0F70@ndsu.edu> I have seen traffic from Microsoft in Europe to single hosts on our campus that seemed to be unusually (high bps) and long. I don?t recall if the few multiple hosts I noticed this on over time were only on our campus wifi. If not perhaps the common factor is longer latency? Both connects over wireless and connections from Europe to the US would have longer latency. Perhaps this longer latency combined with some other factor is triggering a but in modern TCP Congestion Control algorithms? This mentions that there have been bugs in TCP Congestion Control algorithm implementations. Perhaps there could be other bugs that result in the descried issue? https://www.microsoft.com/en-us/research/wp-content/uploads/2016/08/ms_feb07_eval.ppt.pdf I have seen cases on our campus where too small buffers on an ethernet switch caused a Linux TCP Congestion Control algorithm to act badly resulting in slower downloads than a simple algorithm that depended on dropped packets rather than trying to determine window sizes etc. The fix in that case was to increase the buffer size. Of course buffer bloat is also known to play havoc with TCP Congestion Control algorithms. Just wondering if some combination of higher latency and another unknown variable or just a bug might cause a TCP Congestion Control algorithm to think it can safely try to increase the transmit rate? > On Sep 21, 2016, at 8:29 PM, Mike Hammett wrote: > > Thanks Marty. I have only experienced this on my network once and it was directly with Microsoft, so I haven't done much until a couple days ago when I started this campaign. I don't know if anyone else has brought this to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Martin Hannigan" > To: "Mike Hammett" > Cc: "NANOG" > Sent: Wednesday, September 21, 2016 8:19:35 PM > Subject: Re: CDN Overload? > > > > > > Mike, > > > I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > Best, > > > Martin Hannigan > AS 20940 // AS 32787 > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < nanog at ics-il.net > wrote: > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing > > I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" < nanog at ics-il.net > > To: "NANOG" < nanog at nanog.org > > Sent: Wednesday, September 21, 2016 9:08:55 AM > Subject: Re: CDN Overload? > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" < nanog at ics-il.net > > To: "NANOG" < nanog at nanog.org > > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > > > > > --- Bruce Curtis bruce.curtis at ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University From NickFarr at pacbell.net Wed Sep 21 20:11:20 2016 From: NickFarr at pacbell.net (Nick Farr) Date: Wed, 21 Sep 2016 13:11:20 -0700 Subject: Domain renawals In-Reply-To: References: Message-ID: <006701d21444$52e5f6e0$f8b1e4a0$@net> I got sucked into Network Solutions for years over a bunch of domains and procrastinated moving them - but they *BLOOOWWW* big time. Get away from them, you'll be glad you did. It's been a huge relief switching to gandi.net for me, but then I'm not using them for much beyond basic services, YMMV, etc. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Ishmael Rufus Sent: Wednesday, September 21, 2016 9:17 AM To: Jeff Jones Cc: NANOG Subject: Re: Domain renawals $9.88 for commercial domains seems under the average from what I've seen from other registrars On Mon, Sep 19, 2016 at 12:19 PM, Jeff Jones wrote: > Hello All, > > Sorry if this is low level. But are people sick of registrars jacking > up prices? Who is the cheapest and most reliable? I have been using > whois.com , networksolutions.com and am looking for input on who is > cheap, secure, reliable registrar. Thanks for your input. > > ~Jeff > From allan at allan.org Thu Sep 22 02:47:59 2016 From: allan at allan.org (Allan Liska) Date: Wed, 21 Sep 2016 22:47:59 -0400 Subject: Domain renawals In-Reply-To: References: <20160922013555.90778.qmail@ary.lan> Message-ID: <5cda1d7731e9bb69378a518aea9cf7ab@smtp.hushmail.com> I'll throw in a recommendation for Dyn. The reliability and features are excellent and I love their support. allan > On Sep 21, 2016, at 10:10 PM, Justin Paine via NANOG wrote: > > I've had quite good luck with: Gandi, Hover, 101domains, and Google > Domains -- depending on which cc/TLDs you're looking for. > > ____________ > Justin Paine > Head of Trust & Safety > CloudFlare Inc. > PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D > > > On Wed, Sep 21, 2016 at 6:35 PM, John Levine wrote: >>> For domain registration I found that joining the GoDaddy Domain Club >>> ( $120/year or less if you pay ahead for multiple years [1] ) ... >> >> There's a lot of registrars with prepay discounts. Gandi's domains >> are cheaper if you prepay $600, a lot cheaper if you prepay $2000. >> >> R's, >> John > From nanog at ics-il.net Thu Sep 22 23:08:53 2016 From: nanog at ics-il.net (Mike Hammett) Date: Thu, 22 Sep 2016 18:08:53 -0500 (CDT) Subject: CDN Overload? In-Reply-To: <35119344-8731-4C6F-A561-E2E7B43D0F70@ndsu.edu> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> <35119344-8731-4C6F-A561-E2E7B43D0F70@ndsu.edu> Message-ID: <590647901.571587.1474585733789.JavaMail.zimbra@ics-il.net> Do we have any contacts at Microsoft that we can talk to about this? This time around, they are the common denominator. I know people have been complaining about this for longer than Windows 10 has been out, so there must be some other reasons why other parties we are to blame. -----Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP ----- Original Message ----- From: Bruce Curtis To: Mike Hammett Cc: Martin Hannigan , NANOG Sent: Thu, 22 Sep 2016 16:28:17 -0500 (CDT) Subject: Re: CDN Overload? I have seen traffic from Microsoft in Europe to single hosts on our campus that seemed to be unusually (high bps) and long. I don?t recall if the few multiple hosts I noticed this on over time were only on our campus wifi. If not perhaps the common factor is longer latency? Both connects over wireless and connections from Europe to the US would have longer latency. Perhaps this longer latency combined with some other factor is triggering a but in modern TCP Congestion Control algorithms? This mentions that there have been bugs in TCP Congestion Control algorithm implementations. Perhaps there could be other bugs that result in the descried issue? https://www.microsoft.com/en-us/research/wp-content/uploads/2016/08/ms_feb07_eval.ppt.pdf I have seen cases on our campus where too small buffers on an ethernet switch caused a Linux TCP Congestion Control algorithm to act badly resulting in slower downloads than a simple algorithm that depended on dropped packets rather than trying to determine window sizes etc. The fix in that case was to increase the buffer size. Of course buffer bloat is also known to play havoc with TCP Congestion Control algorithms. Just wondering if some combination of higher latency and another unknown variable or just a bug might cause a TCP Congestion Control algorithm to think it can safely try to increase the transmit rate? > On Sep 21, 2016, at 8:29 PM, Mike Hammett wrote: > > Thanks Marty. I have only experienced this on my network once and it was directly with Microsoft, so I haven't done much until a couple days ago when I started this campaign. I don't know if anyone else has brought this to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Martin Hannigan" > To: "Mike Hammett" > Cc: "NANOG" > Sent: Wednesday, September 21, 2016 8:19:35 PM > Subject: Re: CDN Overload? > > > > > > Mike, > > > I will forward to the requisite group for a look. Have you brought this to our attention previously? I don't see anything. If you did, please forward me the ticket numbers or message(s) (peering@ is best) so wee can track down and see if someone already has it in queue. > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > Best, > > > Martin Hannigan > AS 20940 // AS 32787 > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < nanog at ics-il.net > wrote: > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing > > I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" < nanog at ics-il.net > > To: "NANOG" < nanog at nanog.org > > Sent: Wednesday, September 21, 2016 9:08:55 AM > Subject: Re: CDN Overload? > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > ----- Original Message ----- > > From: "Mike Hammett" < nanog at ics-il.net > > To: "NANOG" < nanog at nanog.org > > Sent: Monday, September 19, 2016 12:34:48 PM > Subject: CDN Overload? > > > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > > > > > --- Bruce Curtis bruce.curtis at ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University From hannigan at gmail.com Thu Sep 22 23:29:38 2016 From: hannigan at gmail.com (Martin Hannigan) Date: Thu, 22 Sep 2016 19:29:38 -0400 Subject: CDN Overload? In-Reply-To: <590647901.571587.1474585733789.JavaMail.zimbra@ics-il.net> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> <35119344-8731-4C6F-A561-E2E7B43D0F70@ndsu.edu> <590647901.571587.1474585733789.JavaMail.zimbra@ics-il.net> Message-ID: Mike, I have the right contact there and I'll flag this thread that way in case they havent already seen it. Best, Martin Hannigan AS 20940 // AS 32787 On Thursday, September 22, 2016, Mike Hammett wrote: > Do we have any contacts at Microsoft that we can talk to about this? This > time around, they are the common denominator. I know people have been > complaining about this for longer than Windows 10 has been out, so there > must be some other reasons why other parties we are to blame. > > -----Mike HammettIntelligent Computing SolutionsMidwest Internet > ExchangeThe Brothers WISP > > ----- Original Message ----- > From: Bruce Curtis > > To: Mike Hammett > > Cc: Martin Hannigan >, NANOG < > nanog at nanog.org > > Sent: Thu, 22 Sep 2016 16:28:17 -0500 (CDT) > Subject: Re: CDN Overload? > > > I have seen traffic from Microsoft in Europe to single hosts on our > campus that seemed to be unusually (high bps) and long. > > I don?t recall if the few multiple hosts I noticed this on over time > were only on our campus wifi. > > If not perhaps the common factor is longer latency? Both connects over > wireless and connections from Europe to the US would have longer latency. > > Perhaps this longer latency combined with some other factor is > triggering a but in modern TCP Congestion Control algorithms? > > > > This mentions that there have been bugs in TCP Congestion Control > algorithm implementations. Perhaps there could be other bugs that result > in the descried issue? > > https://www.microsoft.com/en-us/research/wp-content/ > uploads/2016/08/ms_feb07_eval.ppt.pdf > > > I have seen cases on our campus where too small buffers on an ethernet > switch caused a Linux TCP Congestion Control algorithm to act badly > resulting in slower downloads than a simple algorithm that depended on > dropped packets rather than trying to determine window sizes etc. The fix > in that case was to increase the buffer size. Of course buffer bloat is > also known to play havoc with TCP Congestion Control algorithms. Just > wondering if some combination of higher latency and another unknown > variable or just a bug might cause a TCP Congestion Control algorithm to > think it can safely try to increase the transmit rate? > > > > On Sep 21, 2016, at 8:29 PM, Mike Hammett > wrote: > > > > Thanks Marty. I have only experienced this on my network once and it was > directly with Microsoft, so I haven't done much until a couple days ago > when I started this campaign. I don't know if anyone else has brought this > to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Martin Hannigan" > > > To: "Mike Hammett" > > > Cc: "NANOG" > > > Sent: Wednesday, September 21, 2016 8:19:35 PM > > Subject: Re: CDN Overload? > > > > > > > > > > > > Mike, > > > > > > I will forward to the requisite group for a look. Have you brought this > to our attention previously? I don't see anything. If you did, please > forward me the ticket numbers or message(s) (peering@ is best) so wee can > track down and see if someone already has it in queue. > > > > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > > > > Best, > > > > > > Martin Hannigan > > AS 20940 // AS 32787 > > > > > > > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < nanog at ics-il.net > > wrote: > > > > > > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5A > bYUV8CDxGwLSm8/edit?usp=sharing > > > > I have made the anonymized answers public. This will obviously have some > bias to it given that I mostly know fixed wireless operators, but I'm > hoping this gets some good distribution to catch more platforms. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Mike Hammett" < nanog at ics-il.net > > > To: "NANOG" < nanog at nanog.org > > > Sent: Wednesday, September 21, 2016 9:08:55 AM > > Subject: Re: CDN Overload? > > > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > > > I have made this into a Google Form to make it easier to track compared > to randomly formatted responses on multiple mailing lists, Facebook Groups, > etc. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Mike Hammett" < nanog at ics-il.net > > > To: "NANOG" < nanog at nanog.org > > > Sent: Monday, September 19, 2016 12:34:48 PM > > Subject: CDN Overload? > > > > > > I participate on a few other mailing lists focused on eyeball networks. > For a couple years I've been hearing complaints from this CDN or that CDN > was behaving badly. It's been severely ramping up the past few months. > There have been some wild allegations, but I would like to develop a bit > more standardized evidence collection. Initially LimeLight was the only > culprit, but recently it has been Microsoft as well. I'm not sure if there > have been any others. > > > > The principal complaint is that upstream of whatever is doing the rate > limiting for a given customer there is significantly more capacity being > utilized than the customer has purchased. This could happen briefly as TCP > adjusts to the capacity limitation, but in some situations this has > persisted for days at a time. I'll list out a few situations as best as I > can recall them. Some of these may even be merges of a couple situations. > The point is to show the general issue and develop a better process for > collecting what exactly is happening at the time and how to address it. > > > > One situation had approximately 45 megabit/s of capacity being used up > by a customer that had a 1.5 megabit/s plan. All other traffic normally > held itself within the 1.5 megabit/s, but this particular CDN sent > excessively more for extended periods of time. > > > > An often occurrence has someone with a single digit megabit/s limitation > consuming 2x - 3x more than their plan on the other side of the rate > limiter. > > > > Last month on my own network I saw someone with 2x - 3x being consumed > upstream and they had *190* connections downloading said data from > Microsoft. > > > > The past week or two I've been hearing of people only having a single > connection downloading at more than their plan rate. > > > > > > These situations effectively shut out all other Internet traffic to that > customer or even portion of the network for low capacity NLOS areas. It's a > DoS caused by downloads. What happened to the days of MS BITS and you > didn't even notice the download happening? A lot of these guys think that > the CDNs are just a pile of dicks looking to ruin everyone's day and I'm > certain that there are at least a couple people at each CDN that aren't > that way. ;-) > > > > > > > > > > Lots of rambling, sure. What do I need to have these guys collect as > evidence of a problem and who should they send it to? > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > > > > > > > > > > > --- > Bruce Curtis bruce.curtis at ndsu.edu > Certified NetAnalyst II 701-231-8527 > North Dakota State University > > > > > From nanog at ics-il.net Thu Sep 22 23:34:41 2016 From: nanog at ics-il.net (Mike Hammett) Date: Thu, 22 Sep 2016 18:34:41 -0500 (CDT) Subject: CDN Overload? In-Reply-To: References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> <1372495577.13662.1474466934846.JavaMail.mhammett@ThunderFuck> <1842564151.15662.1474504249587.JavaMail.mhammett@ThunderFuck> <270408390.15876.1474507785419.JavaMail.mhammett@ThunderFuck> <35119344-8731-4C6F-A561-E2E7B43D0F70@ndsu.edu> <590647901.571587.1474585733789.JavaMail.zimbra@ics-il.net> Message-ID: <952207165.571666.1474587281847.JavaMail.zimbra@ics-il.net> Thanks. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: Martin Hannigan To: Mike Hammett Cc: NANOG Sent: Thu, 22 Sep 2016 18:29:38 -0500 (CDT) Subject: Re: CDN Overload? Mike, I have the right contact there and I'll flag this thread that way in case they havent already seen it. Best, Martin Hannigan AS 20940 // AS 32787 On Thursday, September 22, 2016, Mike Hammett wrote: > Do we have any contacts at Microsoft that we can talk to about this? This > time around, they are the common denominator. I know people have been > complaining about this for longer than Windows 10 has been out, so there > must be some other reasons why other parties we are to blame. > > -----Mike HammettIntelligent Computing SolutionsMidwest Internet > ExchangeThe Brothers WISP > > ----- Original Message ----- > From: Bruce Curtis > > To: Mike Hammett > > Cc: Martin Hannigan >, NANOG < > nanog at nanog.org > > Sent: Thu, 22 Sep 2016 16:28:17 -0500 (CDT) > Subject: Re: CDN Overload? > > > I have seen traffic from Microsoft in Europe to single hosts on our > campus that seemed to be unusually (high bps) and long. > > I don?t recall if the few multiple hosts I noticed this on over time > were only on our campus wifi. > > If not perhaps the common factor is longer latency? Both connects over > wireless and connections from Europe to the US would have longer latency. > > Perhaps this longer latency combined with some other factor is > triggering a but in modern TCP Congestion Control algorithms? > > > > This mentions that there have been bugs in TCP Congestion Control > algorithm implementations. Perhaps there could be other bugs that result > in the descried issue? > > https://www.microsoft.com/en-us/research/wp-content/ > uploads/2016/08/ms_feb07_eval.ppt.pdf > > > I have seen cases on our campus where too small buffers on an ethernet > switch caused a Linux TCP Congestion Control algorithm to act badly > resulting in slower downloads than a simple algorithm that depended on > dropped packets rather than trying to determine window sizes etc. The fix > in that case was to increase the buffer size. Of course buffer bloat is > also known to play havoc with TCP Congestion Control algorithms. Just > wondering if some combination of higher latency and another unknown > variable or just a bug might cause a TCP Congestion Control algorithm to > think it can safely try to increase the transmit rate? > > > > On Sep 21, 2016, at 8:29 PM, Mike Hammett > wrote: > > > > Thanks Marty. I have only experienced this on my network once and it was > directly with Microsoft, so I haven't done much until a couple days ago > when I started this campaign. I don't know if anyone else has brought this > to anyone's attention. I just sent an e-mail to Owen when I saw yours. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Martin Hannigan" > > > To: "Mike Hammett" > > > Cc: "NANOG" > > > Sent: Wednesday, September 21, 2016 8:19:35 PM > > Subject: Re: CDN Overload? > > > > > > > > > > > > Mike, > > > > > > I will forward to the requisite group for a look. Have you brought this > to our attention previously? I don't see anything. If you did, please > forward me the ticket numbers or message(s) (peering@ is best) so wee can > track down and see if someone already has it in queue. > > > > > > Jared alluded to fasttcp a few emails ago. Astute man. > > > > > > Best, > > > > > > Martin Hannigan > > AS 20940 // AS 32787 > > > > > > > > > > > > On Sep 21, 2016, at 14:30, Mike Hammett < nanog at ics-il.net > > wrote: > > > > > > > > > > https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5A > bYUV8CDxGwLSm8/edit?usp=sharing > > > > I have made the anonymized answers public. This will obviously have some > bias to it given that I mostly know fixed wireless operators, but I'm > hoping this gets some good distribution to catch more platforms. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Mike Hammett" < nanog at ics-il.net > > > To: "NANOG" < nanog at nanog.org > > > Sent: Wednesday, September 21, 2016 9:08:55 AM > > Subject: Re: CDN Overload? > > > > https://goo.gl/forms/LvgFRsMdNdI8E9HF3 > > > > I have made this into a Google Form to make it easier to track compared > to randomly formatted responses on multiple mailing lists, Facebook Groups, > etc. > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > ----- Original Message ----- > > > > From: "Mike Hammett" < nanog at ics-il.net > > > To: "NANOG" < nanog at nanog.org > > > Sent: Monday, September 19, 2016 12:34:48 PM > > Subject: CDN Overload? > > > > > > I participate on a few other mailing lists focused on eyeball networks. > For a couple years I've been hearing complaints from this CDN or that CDN > was behaving badly. It's been severely ramping up the past few months. > There have been some wild allegations, but I would like to develop a bit > more standardized evidence collection. Initially LimeLight was the only > culprit, but recently it has been Microsoft as well. I'm not sure if there > have been any others. > > > > The principal complaint is that upstream of whatever is doing the rate > limiting for a given customer there is significantly more capacity being > utilized than the customer has purchased. This could happen briefly as TCP > adjusts to the capacity limitation, but in some situations this has > persisted for days at a time. I'll list out a few situations as best as I > can recall them. Some of these may even be merges of a couple situations. > The point is to show the general issue and develop a better process for > collecting what exactly is happening at the time and how to address it. > > > > One situation had approximately 45 megabit/s of capacity being used up > by a customer that had a 1.5 megabit/s plan. All other traffic normally > held itself within the 1.5 megabit/s, but this particular CDN sent > excessively more for extended periods of time. > > > > An often occurrence has someone with a single digit megabit/s limitation > consuming 2x - 3x more than their plan on the other side of the rate > limiter. > > > > Last month on my own network I saw someone with 2x - 3x being consumed > upstream and they had *190* connections downloading said data from > Microsoft. > > > > The past week or two I've been hearing of people only having a single > connection downloading at more than their plan rate. > > > > > > These situations effectively shut out all other Internet traffic to that > customer or even portion of the network for low capacity NLOS areas. It's a > DoS caused by downloads. What happened to the days of MS BITS and you > didn't even notice the download happening? A lot of these guys think that > the CDNs are just a pile of dicks looking to ruin everyone's day and I'm > certain that there are at least a couple people at each CDN that aren't > that way. ;-) > > > > > > > > > > Lots of rambling, sure. What do I need to have these guys collect as > evidence of a problem and who should they send it to? > > > > > > > > > > ----- > > Mike Hammett > > Intelligent Computing Solutions > > > > Midwest Internet Exchange > > > > The Brothers WISP > > > > > > > > > > > > > > --- > Bruce Curtis bruce.curtis at ndsu.edu > Certified NetAnalyst II 701-231-8527 > North Dakota State University > > > > > From rsk at gsp.org Fri Sep 23 12:35:11 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Fri, 23 Sep 2016 08:35:11 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: <87mvj3slyr.fsf@mid.deneb.enyo.de> References: <20160916131246.GP29651@dilbert.slimey.org> <20160918130703.GA17415@gsp.org> <87k2e91fch.fsf@mid.deneb.enyo.de> <20160919175324.GA27679@gsp.org> <87mvj3slyr.fsf@mid.deneb.enyo.de> Message-ID: <20160923123511.GA31214@gsp.org> On Mon, Sep 19, 2016 at 09:55:56PM +0200, Florian Weimer wrote: > Github users create several orders of magnitude more SSH connections > [snip] Ah. I didn't know that. Thanks! > Sure, and people already do this, and are not very flexible about it. > Support staff isn't briefed, and claim they do such stochastic > behavior adjustment across all (server) products, which I find > difficult to believe. You're right: those are serious drawbacks. If folks are going to do this, then they need to do it right, which means making sure everyone is in the loop and making sure that support staff are clueful/diligent enough to investigate -- or at least hand off to someone who'll investigate. This stuff works but only if you're adaptive/flexible and willing to learn and adjust on an ongoing basis. > I'm worried that this leads to a future where tunnelling everything > over HTTP(S) is no longer sufficient. You have to make it look like a > web server or browser, too. Everything else risks triggering > automated countermeasures. And as someone who constantly beats the "Internet != web" drum, I second this. Marginalizing other protocols doesn't serve us well in short term (it breaks things) or the long term (it stifles innovation). ---rsk From rsk at gsp.org Fri Sep 23 12:39:09 2016 From: rsk at gsp.org (Rich Kulawiec) Date: Fri, 23 Sep 2016 08:39:09 -0400 Subject: PlayStationNetwork blocking of CGNAT public addresses In-Reply-To: References: Message-ID: <20160923123909.GB31214@gsp.org> On Thu, Sep 22, 2016 at 02:31:12PM +0200, Alexander Maassen wrote: > Maybe its time then for a global accepted, unified way to send/report abuse??? There are -- see Valdis's followup. But there's still no viable substitute for a working abuse@ address with clueful eyeballs on the other side of it. Every responsible and professional operation on this planet has that. The really good ones learn from what shows up there and pro-actively deal with abuse issues before anyone else is bothered by them, which not only makes them better netizens but reduces the volume of incoming complaints. ---rsk From shawnl at up.net Fri Sep 23 12:50:06 2016 From: shawnl at up.net (Shawn L) Date: Fri, 23 Sep 2016 08:50:06 -0400 (EDT) Subject: =?utf-8?Q?Manage_Outage_Notifications=3F?= Message-ID: <1474635006.080312898@upnet.mymailsrvr.com> What are people using to manage / send their outage notifications? We're currently using a mostly manual process to identify customers that need to be aware of an outage and send out e-mail at $dayjob. Looking for a way to automate it more. I'd prefer something open source, but that's not a requirement. Thanks From bill at herrin.us Fri Sep 23 13:13:02 2016 From: bill at herrin.us (William Herrin) Date: Fri, 23 Sep 2016 09:13:02 -0400 Subject: Manage Outage Notifications? In-Reply-To: <1474635006.080312898@upnet.mymailsrvr.com> References: <1474635006.080312898@upnet.mymailsrvr.com> Message-ID: On Fri, Sep 23, 2016 at 8:50 AM, Shawn L wrote: > What are people using to manage / send their outage notifications? Hi Shawn, I've been very impressed lately with https://www.pagerduty.com/ Nice control of a staff rotation. Distribution to multiple groups depending on area of responsibility. Very customizable escalation. Android and iPhone apps in addition to SMS and voice call support. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From shortdudey123 at gmail.com Fri Sep 23 17:58:44 2016 From: shortdudey123 at gmail.com (Grant Ridder) Date: Fri, 23 Sep 2016 10:58:44 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Message-ID: Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ "Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size." -Grant From cscora at apnic.net Fri Sep 23 18:01:30 2016 From: cscora at apnic.net (Routing Analysis Role Account) Date: Sat, 24 Sep 2016 04:01:30 +1000 (AEST) Subject: Weekly Routing Table Report Message-ID: <20160923180130.7F888BF120@thyme.apnic.net> This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats at lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith . Routing Table Report 04:00 +10GMT Sat 24 Sep, 2016 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 611401 Prefixes after maximum aggregation (per Origin AS): 220201 Deaggregation factor: 2.78 Unique aggregates announced (without unneeded subnets): 298301 Total ASes present in the Internet Routing Table: 54892 Prefixes per ASN: 11.14 Origin-only ASes present in the Internet Routing Table: 36377 Origin ASes announcing only one prefix: 15380 Transit ASes present in the Internet Routing Table: 6523 Transit-only ASes present in the Internet Routing Table: 171 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 54 Max AS path prepend of ASN ( 55644) 51 Prefixes from unregistered ASNs in the Routing Table: 65 Unregistered ASNs in the Routing Table: 20 Number of 32-bit ASNs allocated by the RIRs: 15526 Number of 32-bit ASNs visible in the Routing Table: 11992 Prefixes from 32-bit ASNs in the Routing Table: 47864 Number of bogon 32-bit ASNs visible in the Routing Table: 102 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 342 Number of addresses announced to Internet: 2828439524 Equivalent to 168 /8s, 150 /16s and 143 /24s Percentage of available address space announced: 76.4 Percentage of allocated address space announced: 76.4 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.2 Total number of prefixes smaller than registry allocations: 198636 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 156403 Total APNIC prefixes after maximum aggregation: 42921 APNIC Deaggregation factor: 3.64 Prefixes being announced from the APNIC address blocks: 170011 Unique aggregates announced from the APNIC address blocks: 69184 APNIC Region origin ASes present in the Internet Routing Table: 5191 APNIC Prefixes per ASN: 32.75 APNIC Region origin ASes announcing only one prefix: 1148 APNIC Region transit ASes present in the Internet Routing Table: 947 Average APNIC Region AS path length visible: 4.5 Max APNIC Region AS path length visible: 54 Number of APNIC region 32-bit ASNs visible in the Routing Table: 2389 Number of APNIC addresses announced to Internet: 759948228 Equivalent to 45 /8s, 75 /16s and 227 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 184481 Total ARIN prefixes after maximum aggregation: 89596 ARIN Deaggregation factor: 2.06 Prefixes being announced from the ARIN address blocks: 190286 Unique aggregates announced from the ARIN address blocks: 88489 ARIN Region origin ASes present in the Internet Routing Table: 16188 ARIN Prefixes per ASN: 11.75 ARIN Region origin ASes announcing only one prefix: 5689 ARIN Region transit ASes present in the Internet Routing Table: 1705 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 1477 Number of ARIN addresses announced to Internet: 1107456736 Equivalent to 66 /8s, 2 /16s and 114 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 146000 Total RIPE prefixes after maximum aggregation: 71898 RIPE Deaggregation factor: 2.03 Prefixes being announced from the RIPE address blocks: 156352 Unique aggregates announced from the RIPE address blocks: 96559 RIPE Region origin ASes present in the Internet Routing Table: 18136 RIPE Prefixes per ASN: 8.62 RIPE Region origin ASes announcing only one prefix: 7827 RIPE Region transit ASes present in the Internet Routing Table: 3039 Average RIPE Region AS path length visible: 4.4 Max RIPE Region AS path length visible: 27 Number of RIPE region 32-bit ASNs visible in the Routing Table: 5056 Number of RIPE addresses announced to Internet: 708744960 Equivalent to 42 /8s, 62 /16s and 151 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 61851 Total LACNIC prefixes after maximum aggregation: 12484 LACNIC Deaggregation factor: 4.95 Prefixes being announced from the LACNIC address blocks: 77626 Unique aggregates announced from the LACNIC address blocks: 37425 LACNIC Region origin ASes present in the Internet Routing Table: 2474 LACNIC Prefixes per ASN: 31.38 LACNIC Region origin ASes announcing only one prefix: 544 LACNIC Region transit ASes present in the Internet Routing Table: 579 Average LACNIC Region AS path length visible: 4.8 Max LACNIC Region AS path length visible: 26 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 2807 Number of LACNIC addresses announced to Internet: 170771520 Equivalent to 10 /8s, 45 /16s and 196 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-266652 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 14736 Total AfriNIC prefixes after maximum aggregation: 3292 AfriNIC Deaggregation factor: 4.48 Prefixes being announced from the AfriNIC address blocks: 16784 Unique aggregates announced from the AfriNIC address blocks: 6330 AfriNIC Region origin ASes present in the Internet Routing Table: 733 AfriNIC Prefixes per ASN: 22.90 AfriNIC Region origin ASes announcing only one prefix: 172 AfriNIC Region transit ASes present in the Internet Routing Table: 182 Average AfriNIC Region AS path length visible: 4.6 Max AfriNIC Region AS path length visible: 18 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 263 Number of AfriNIC addresses announced to Internet: 81179648 Equivalent to 4 /8s, 214 /16s and 180 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5558 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3525 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 3014 11138 984 KIXS-AS-KR Korea Telecom, KR 17974 2943 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2694 1497 532 BSNL-NIB National Internet Backbone, IN 9808 2246 8781 42 CMNET-GD Guangdong Mobile Communication 4755 2053 429 222 TATACOMM-AS TATA Communications formerl 4808 1757 2289 524 CHINA169-BJ China Unicom Beijing Provin 45899 1601 1235 89 VNPT-AS-VN VNPT Corp, VN 24560 1548 515 223 AIRTELBROADBAND-AS-AP Bharti Airtel Ltd Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 22773 3515 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 6389 2221 3671 41 BELLSOUTH-NET-BLK - BellSouth.net Inc., 18566 2194 405 110 MEGAPATH5-US - MegaPath Corporation, US 20115 1940 1982 405 CHARTER-NET-HKY-NC - Charter Communicat 30036 1752 339 291 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 209 1730 5071 664 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 6983 1676 848 227 ITCDELTA - Earthlink, Inc., US 701 1625 10693 678 UUNET - MCI Communications Services, In 16509 1403 2534 472 AMAZON-02 - Amazon.com, Inc., US 7018 1365 20092 1005 ATT-INTERNET4 - AT&T Services, Inc., US Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3329 169 15 ALJAWWALSTC-AS , SA 20940 2816 1069 1999 AKAMAI-ASN1 , US 34984 1968 325 361 TELLCOM-AS , TR 12479 1343 1018 46 UNI2-AS , ES 13188 1236 99 57 BANKINFORM-AS , UA 8551 1206 377 46 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 6849 1144 355 21 UKRTELNET , UA 8402 1089 544 15 CORBINA-AS Russia, RU 9198 934 352 25 KAZTELECOM-AS , KZ 12389 925 1113 401 ROSTELECOM-AS , RU Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 10620 3484 540 184 Telmex Colombia S.A., CO 8151 2284 3396 558 Uninet S.A. de C.V., MX 7303 1543 955 245 Telecom Argentina S.A., AR 6503 1404 437 55 Axtel, S.A.B. de C.V., MX 11830 1326 367 57 Instituto Costarricense de Electricidad 6147 1078 377 27 Telefonica del Peru S.A.A., PE 7738 994 1882 40 Telemar Norte Leste S.A., BR 3816 965 468 202 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 28573 934 2203 176 CLARO S.A., BR 11172 907 125 76 Alestra, S. de R.L. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1179 402 50 LINKdotNET-AS, EG 36903 684 344 115 MT-MPLS, MA 37611 666 51 5 Afrihost, ZA 36992 547 1373 25 ETISALAT-MISR, EG 8452 539 1472 15 TE-AS TE-AS, EG 37492 384 250 69 ORANGE-TN, TN 24835 349 611 17 RAYA-AS, EG 29571 318 37 12 CITelecom-AS, CI 15399 298 35 7 WANANCHI-KE, KE 2018 269 327 74 TENET-1, ZA Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5558 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3525 382 250 TPG-INTERNET-AP TPG Telecom Limited, AU 22773 3515 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3484 540 184 Telmex Colombia S.A., CO 39891 3329 169 15 ALJAWWALSTC-AS , SA 4766 3014 11138 984 KIXS-AS-KR Korea Telecom, KR 17974 2943 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 20940 2816 1069 1999 AKAMAI-ASN1 , US 9829 2694 1497 532 BSNL-NIB National Internet Backbone, IN 8151 2284 3396 558 Uninet S.A. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 22773 3515 3370 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 39891 3329 3314 ALJAWWALSTC-AS , SA 10620 3484 3300 Telmex Colombia S.A., CO 7545 3525 3275 TPG-INTERNET-AP TPG Telecom Limited, AU 17974 2943 2865 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9808 2246 2204 CMNET-GD Guangdong Mobile Communication 6389 2221 2180 BELLSOUTH-NET-BLK - BellSouth.net Inc., 9829 2694 2162 BSNL-NIB National Internet Backbone, IN 18566 2194 2084 MEGAPATH5-US - MegaPath Corporation, US 4766 3014 2030 KIXS-AS-KR Korea Telecom, KR Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65001 PRIVATE 5.143.176.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.200.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.208.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.216.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65000 PRIVATE 31.219.177.0/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65000 PRIVATE 31.219.177.128/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65412 PRIVATE 41.89.7.0/24 36866 JTL, KE 65512 PRIVATE 45.252.244.0/24 45899 VNPT-AS-VN VNPT Corp, VN Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 23.249.144.0/20 40430 COLO4JAX-AS - colo4jax, LLC, US 24.49.144.0/20 30164 USACOMMUNICATIONS-1 - USA COMPANIES, LP 27.100.7.0/24 56096 UNKNOWN 41.73.1.0/24 37004 -Reserved AS-, ZZ 41.73.2.0/24 37004 -Reserved AS-, ZZ 41.73.3.0/24 37004 -Reserved AS-, ZZ 41.73.4.0/24 37004 -Reserved AS-, ZZ 41.73.5.0/24 37004 -Reserved AS-, ZZ 41.73.6.0/24 37004 -Reserved AS-, ZZ 41.73.7.0/24 37004 -Reserved AS-, ZZ Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:16 /9:13 /10:36 /11:103 /12:273 /13:523 /14:1052 /15:1773 /16:13150 /17:7833 /18:13012 /19:25408 /20:38759 /21:40323 /22:68143 /23:59710 /24:339823 /25:504 /26:475 /27:338 /28:55 /29:32 /30:13 /31:1 /32:33 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 39891 2896 3329 ALJAWWALSTC-AS , SA 22773 2739 3515 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2086 2194 MEGAPATH5-US - MegaPath Corporation, US 30036 1568 1752 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 6389 1418 2221 BELLSOUTH-NET-BLK - BellSouth.net Inc., 10620 1408 3484 Telmex Colombia S.A., CO 6983 1325 1676 ITCDELTA - Earthlink, Inc., US 34984 1253 1968 TELLCOM-AS , TR 11492 1206 1304 CABLEONE - CABLE ONE, INC., US 13188 1011 1236 BANKINFORM-AS , UA Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1644 2:780 4:23 5:2174 6:31 8:991 12:1805 13:43 14:1742 15:46 16:2 17:97 18:126 20:50 22:1 23:1817 24:1803 27:2339 31:1783 32:84 33:4 35:5 36:328 37:2316 38:1247 39:36 40:101 41:2952 42:454 43:1836 44:47 45:2141 46:2545 47:98 49:1213 50:907 51:13 52:551 53:2 54:346 55:7 56:7 57:41 58:1570 59:1009 60:378 61:1823 62:1511 63:1927 64:4556 65:2174 66:4277 67:2209 68:1120 69:3263 70:1283 71:556 72:2058 74:2590 75:400 76:413 77:1380 78:1214 79:911 80:1298 81:1397 82:982 83:705 84:858 85:1566 86:483 87:1128 88:543 89:2098 90:231 91:6087 92:955 93:2373 94:2438 95:2487 96:558 97:341 98:959 99:89 100:148 101:1153 103:12333 104:2482 105:108 106:434 107:1428 108:786 109:2257 110:1271 111:1635 112:1054 113:1151 114:1084 115:1684 116:1675 117:1562 118:2038 119:1576 120:933 121:1121 122:2267 123:2039 124:1569 125:1824 128:666 129:439 130:417 131:1369 132:612 133:175 134:490 135:196 136:376 137:405 138:1885 139:430 140:579 141:462 142:664 143:955 144:729 145:164 146:945 147:659 148:1381 149:536 150:657 151:879 152:664 153:297 154:695 155:950 156:525 157:514 158:386 159:1239 160:510 161:735 162:2403 163:555 164:770 165:1135 166:335 167:1188 168:2187 169:667 170:1955 171:251 172:708 173:1741 174:760 175:674 176:1723 177:3940 178:2240 179:1179 180:2149 181:1869 182:2057 183:1001 184:824 185:7474 186:3129 187:2118 188:2204 189:1757 190:7673 191:1272 192:9020 193:5736 194:4450 195:3844 196:1773 197:1150 198:5581 199:5758 200:7159 201:3654 202:10113 203:9836 204:4514 205:2746 206:2980 207:3075 208:4046 209:3861 210:3886 211:2039 212:2723 213:2345 214:865 215:70 216:5775 217:1973 218:810 219:607 220:1635 221:883 222:699 223:1306 End of report From nanog at ics-il.net Fri Sep 23 18:01:49 2016 From: nanog at ics-il.net (Mike Hammett) Date: Fri, 23 Sep 2016 13:01:49 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> I believe the article says they were being hosted for free. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Grant Ridder" To: nanog at nanog.org Sent: Friday, September 23, 2016 12:58:44 PM Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ "Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size." -Grant From cma at cmadams.net Fri Sep 23 18:02:55 2016 From: cma at cmadams.net (Chris Adams) Date: Fri, 23 Sep 2016 13:02:55 -0500 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <20160923180255.GA2284@cmadams.net> Once upon a time, Grant Ridder said: > Didn't realize Akamai kicked out or disabled customers Any business is likely to kick out customers that cost them much more than they are being paid (under relevant contract terms of course). Since his blog was being hosted for free, it isn't surprising that Akamai told him they couldn't do that anymore. It certainly isn't fair to expect Akamai (and their paying customers) to deal with that. -- Chris Adams From alex at alexwacker.com Fri Sep 23 18:02:56 2016 From: alex at alexwacker.com (Alex Wacker) Date: Fri, 23 Sep 2016 11:02:56 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: To be fair, he was getting the service for free. I wouldn?t really call that a paying customer. Still not great from a PR standpoint though. -- Alex Wacker On September 23, 2016 at 2:00:10 PM, Grant Ridder (shortdudey123 at gmail.com) wrote: Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ "Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size." -Grant From simon at slimey.org Fri Sep 23 18:03:57 2016 From: simon at slimey.org (Simon Lockhart) Date: Fri, 23 Sep 2016 19:03:57 +0100 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <20160923180357.GB29651@dilbert.slimey.org> On Fri Sep 23, 2016 at 10:58:44AM -0700, Grant Ridder wrote: > Didn't realize Akamai kicked out or disabled customers They didn't - Krebs has publicly stated that Akamai were providing services "Pro Bono" - and I guess the goodwill ran out :) Simon From JKrejci at usinternet.com Fri Sep 23 18:05:30 2016 From: JKrejci at usinternet.com (Justin Krejci) Date: Fri, 23 Sep 2016 18:05:30 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <3E9C67DA261AC349B60FF3609F5E211D7D764596@USI-2K10EX01-MT.usicorp.usinternet.com> Message-ID: <57e56eed.c4a06b0a.2b706.b1edSMTPIN_ADDED_BROKEN@mx.google.com> If you read the article, it is made clear he was "kicked off" of a free service being provided. He was not a paying customer of Akamai and does not fault Akamai for their decision. ________________________________________ From: Grant Ridder [shortdudey123 at gmail.com] Sent: Friday, September 23, 2016 12:58 PM To: nanog at nanog.org Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ "Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size." -Grant From patrick at ianai.net Fri Sep 23 18:07:19 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Fri, 23 Sep 2016 14:07:19 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: On Sep 23, 2016, at 1:58 PM, Grant Ridder wrote: > > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size.? Even Brian Krebs doesn?t think Akamai kicks out or disables customers. In the article, he admits he is not paying Akamai. Should Akamai have kept protecting Krebs? Maybe. But I do not think anyone should pass judgement on Akamai - unless that person is willing to pick up the tab. (And some of the companies claiming they will pick up the tab are just hoping for PR, since they could not have actually protected Krebs.) -- TTFN, patrick From deleskie at gmail.com Fri Sep 23 18:09:53 2016 From: deleskie at gmail.com (jim deleskie) Date: Fri, 23 Sep 2016 15:09:53 -0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: They were hosting him for free, and like insurance, I can assure you if you are consistently using a service, and not covering the costs of that service you won't be a client for long. This is the basis for AUP/client contracts and have been going back to the days when we all offered only dialup internet. On Fri, Sep 23, 2016 at 3:01 PM, Mike Hammett wrote: > I believe the article says they were being hosted for free. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > Midwest-IX > http://www.midwest-ix.com > > ----- Original Message ----- > > From: "Grant Ridder" > To: nanog at nanog.org > Sent: Friday, September 23, 2016 12:58:44 PM > Subject: Krebs on Security booted off Akamai network after DDoS attack > proves pricey > > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off- > akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size." > > -Grant > > From rubensk at gmail.com Fri Sep 23 18:11:45 2016 From: rubensk at gmail.com (Rubens Kuhl) Date: Fri, 23 Sep 2016 15:11:45 -0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: On Fri, Sep 23, 2016 at 2:58 PM, Grant Ridder wrote: > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off- > akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size." > So much for defending free speech... Rubens From fhr at fhrnet.eu Fri Sep 23 18:15:01 2016 From: fhr at fhrnet.eu (Filip Hruska) Date: Fri, 23 Sep 2016 20:15:01 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160923180255.GA2284@cmadams.net> References: <20160923180255.GA2284@cmadams.net> Message-ID: While we are on topic of DDOS, it looks like it's quite a storm now. According to this WHT post [1], some large server providers were recently attacked, and many are still being attacked with quite a large bandwidth, ie 1Tbps attacks against OVH. [2], [3] Regards, Filip [1] http://www.webhostingtalk.com/showthread.php?t=1599694 [2] https://twitter.com/olesovhcom/status/778019962036314112 [3] https://twitter.com/olesovhcom/status/778830571677978624 On 23.9.2016 20:02, Chris Adams wrote: > Once upon a time, Grant Ridder said: >> Didn't realize Akamai kicked out or disabled customers > > Any business is likely to kick out customers that cost them much more > than they are being paid (under relevant contract terms of course). > Since his blog was being hosted for free, it isn't surprising that > Akamai told him they couldn't do that anymore. > > It certainly isn't fair to expect Akamai (and their paying customers) to > deal with that. > From mel at beckman.org Fri Sep 23 18:16:11 2016 From: mel at beckman.org (Mel Beckman) Date: Fri, 23 Sep 2016 18:16:11 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <8DCD65CF-8574-44CB-AF46-DD034FCC82FE@beckman.org> My gigabit pipe was also DDOS attacked the same day my name appeared in Brian?s story. -mel > On Sep 23, 2016, at 11:02 AM, Alex Wacker wrote: > > To be fair, he was getting the service for free. I wouldn?t really call > that a paying customer. Still not great from a PR standpoint though. > > -- > Alex Wacker > > > On September 23, 2016 at 2:00:10 PM, Grant Ridder (shortdudey123 at gmail.com) > wrote: > > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size." > > -Grant From daknob.mac at gmail.com Fri Sep 23 18:22:15 2016 From: daknob.mac at gmail.com (DaKnOb) Date: Fri, 23 Sep 2016 21:22:15 +0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <75FBB90E-D9B6-423B-94BE-BE41E4BF7CFD@gmail.com> Well, there?s always Cloudflare and Google that are willing to do it for free. Let?s hope we won?t run out of free providers any time soon.. It?s a nice blog. > On 23 Sep 2016, at 20:58, Grant Ridder wrote: > > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size." > > -Grant From sethm at rollernet.us Fri Sep 23 18:30:57 2016 From: sethm at rollernet.us (Seth Mattinen) Date: Fri, 23 Sep 2016 11:30:57 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: Message-ID: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> On 9/23/16 10:58, Grant Ridder wrote: > Didn't realize Akamai kicked out or disabled customers > http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > > "Security blog Krebs on Security has been taken offline by host Akamai > Technologies following a DDoS attack which reached 665 Gbps in size." So ultimately the DDoS was successful, just in a different way. ~Seth From mike-nanog at tiedyenetworks.com Fri Sep 23 18:45:45 2016 From: mike-nanog at tiedyenetworks.com (Mike) Date: Fri, 23 Sep 2016 11:45:45 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> Message-ID: <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> On 09/23/2016 11:30 AM, Seth Mattinen wrote: > On 9/23/16 10:58, Grant Ridder wrote: >> Didn't realize Akamai kicked out or disabled customers >> http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ >> >> >> "Security blog Krebs on Security has been taken offline by host Akamai >> Technologies following a DDoS attack which reached 665 Gbps in size." > > > So ultimately the DDoS was successful, just in a different way. > > ~Seth > > More technical information about the characteristics of these attacks would be very interesting such as the ultimate sources of the attack traffic (compromised home pc's?), the nature of the traffic (dns / ssdp amplification?), whether it was spoofed source (BCP38-adverse), and whether the recent takedown the vDOS was really complete or if it's likely someone else gained control of the C&C servers that controlled it's assets? Mike- From saper at saper.info Fri Sep 23 18:58:08 2016 From: saper at saper.info (Marcin Cieslak) Date: Fri, 23 Sep 2016 18:58:08 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: On Fri, 23 Sep 2016, jim deleskie wrote: > They were hosting him for free, and like insurance, I can assure you if you > are consistently using a service, and not covering the costs of that > service you won't be a client for long. This is the basis for AUP/client > contracts and have been going back to the days when we all offered only > dialup internet. Does being a victim of a DDoS constitute a breach of AUP? Marcin Cie?lak From haegar at sdinet.de Fri Sep 23 19:15:26 2016 From: haegar at sdinet.de (Sven-Haegar Koch) Date: Fri, 23 Sep 2016 21:15:26 +0200 (CEST) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> Message-ID: On Fri, 23 Sep 2016, Mike wrote: > On 09/23/2016 11:30 AM, Seth Mattinen wrote: > > On 9/23/16 10:58, Grant Ridder wrote: > > > Didn't realize Akamai kicked out or disabled customers > > > http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > > > > > > "Security blog Krebs on Security has been taken offline by host Akamai > > > Technologies following a DDoS attack which reached 665 Gbps in size." > > > > > > So ultimately the DDoS was successful, just in a different way. > > > > ~Seth > > > > > More technical information about the characteristics of these attacks would be > very interesting such as the ultimate sources of the attack traffic > (compromised home pc's?), the nature of the traffic (dns / ssdp > amplification?), whether it was spoofed source (BCP38-adverse), and whether > the recent takedown the vDOS was really complete or if it's likely someone > else gained control of the C&C servers that controlled it's assets? At least for the OVH case there is a bit of info: https://twitter.com/olesovhcom/status/779297257199964160 "This botnet with 145607 cameras/dvr (1-30Mbps per IP) is able to send >1.5Tbps DDoS. Type: tcp/ack, tcp/ack+psh, tcp/syn." c'ya sven-haegar -- Three may keep a secret, if two of them are dead. - Ben F. From justin at cloudflare.com Fri Sep 23 19:16:45 2016 From: justin at cloudflare.com (Justin Paine) Date: Fri, 23 Sep 2016 12:16:45 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: FWIW, we have offered to help. No word so far. We're more than willing to step in front of the cannon pointed his way. ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak wrote: > On Fri, 23 Sep 2016, jim deleskie wrote: > >> They were hosting him for free, and like insurance, I can assure you if you >> are consistently using a service, and not covering the costs of that >> service you won't be a client for long. This is the basis for AUP/client >> contracts and have been going back to the days when we all offered only >> dialup internet. > > Does being a victim of a DDoS constitute a breach of AUP? > > Marcin Cie?lak From patrick at ianai.net Fri Sep 23 19:26:14 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Fri, 23 Sep 2016 15:26:14 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: Is CloudFlare able to filter Layer 7 these days? I was under the impression CloudFlare was not able to do that. There have been a lot of rumors about this attack. Some say reflection, others say Layer 7, others say .. other stuff. If it is Layer 7, how are you going to ?step in front of the cannon?? Would you just pass through all the traffic? I realize Matthew is always happy for publicity (hell, the whole planet is aware of that). But if your system cannot actually do the required task, I?m not sure your company should give you credit for offering a service the user cannot use. -- TTFN, patrick > On Sep 23, 2016, at 3:16 PM, Justin Paine via NANOG wrote: > > FWIW, we have offered to help. No word so far. We're more than willing > to step in front of the cannon pointed his way. > > ____________ > Justin Paine > Head of Trust & Safety > CloudFlare Inc. > PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D > > > On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak wrote: >> On Fri, 23 Sep 2016, jim deleskie wrote: >> >>> They were hosting him for free, and like insurance, I can assure you if you >>> are consistently using a service, and not covering the costs of that >>> service you won't be a client for long. This is the basis for AUP/client >>> contracts and have been going back to the days when we all offered only >>> dialup internet. >> >> Does being a victim of a DDoS constitute a breach of AUP? >> >> Marcin Cie?lak From justin at cloudflare.com Fri Sep 23 19:29:42 2016 From: justin at cloudflare.com (Justin Paine) Date: Fri, 23 Sep 2016 12:29:42 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: We routinely mitigate L7s. Matthew is also on the record saying we've seen and mitigated similar attacks to this one (based on available information about this attack). ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Fri, Sep 23, 2016 at 12:26 PM, Patrick W. Gilmore wrote: > Is CloudFlare able to filter Layer 7 these days? I was under the impression CloudFlare was not able to do that. > > There have been a lot of rumors about this attack. Some say reflection, others say Layer 7, others say .. other stuff. If it is Layer 7, how are you going to ?step in front of the cannon?? Would you just pass through all the traffic? > > I realize Matthew is always happy for publicity (hell, the whole planet is aware of that). But if your system cannot actually do the required task, I?m not sure your company should give you credit for offering a service the user cannot use. > > -- > TTFN, > patrick > >> On Sep 23, 2016, at 3:16 PM, Justin Paine via NANOG wrote: >> >> FWIW, we have offered to help. No word so far. We're more than willing >> to step in front of the cannon pointed his way. >> >> ____________ >> Justin Paine >> Head of Trust & Safety >> CloudFlare Inc. >> PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D >> >> >> On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak wrote: >>> On Fri, 23 Sep 2016, jim deleskie wrote: >>> >>>> They were hosting him for free, and like insurance, I can assure you if you >>>> are consistently using a service, and not covering the costs of that >>>> service you won't be a client for long. This is the basis for AUP/client >>>> contracts and have been going back to the days when we all offered only >>>> dialup internet. >>> >>> Does being a victim of a DDoS constitute a breach of AUP? >>> >>> Marcin Cie?lak > From jk at ip-clear.de Fri Sep 23 19:30:43 2016 From: jk at ip-clear.de (=?utf-8?q?J=C3=B6rg?= Kost) Date: Fri, 23 Sep 2016 21:30:43 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: <789F3795-C416-48CB-ADE5-1AE9CEB6DE64@ip-clear.de> Yes, they do (or advertise): https://support.cloudflare.com/hc/en-us/articles/200170216-How-large-of-a-DDoS-attack-can-CloudFlare-handle- J?rg On 23 Sep 2016, at 21:26, Patrick W. Gilmore wrote: > Is CloudFlare able to filter Layer 7 these days? I was under the > impression CloudFlare was not able to do that. > > There have been a lot of rumors about this attack. Some say > reflection, others say Layer 7, others say .. other stuff. If it is > Layer 7, how are you going to ?step in front of the cannon?? Would > you just pass through all the traffic? > > I realize Matthew is always happy for publicity (hell, the whole > planet is aware of that). But if your system cannot actually do the > required task, I?m not sure your company should give you credit for > offering a service the user cannot use. > > -- > TTFN, > patrick > >> On Sep 23, 2016, at 3:16 PM, Justin Paine via NANOG >> wrote: >> >> FWIW, we have offered to help. No word so far. We're more than >> willing >> to step in front of the cannon pointed his way. >> >> ____________ >> Justin Paine >> Head of Trust & Safety >> CloudFlare Inc. >> PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D >> >> >> On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak >> wrote: >>> On Fri, 23 Sep 2016, jim deleskie wrote: >>> >>>> They were hosting him for free, and like insurance, I can assure >>>> you if you >>>> are consistently using a service, and not covering the costs of >>>> that >>>> service you won't be a client for long. This is the basis for >>>> AUP/client >>>> contracts and have been going back to the days when we all offered >>>> only >>>> dialup internet. >>> >>> Does being a victim of a DDoS constitute a breach of AUP? >>> >>> Marcin Cie?lak > From hugo at slabnet.com Fri Sep 23 21:24:53 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Fri, 23 Sep 2016 14:24:53 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> Message-ID: <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> On September 23, 2016 12:15:26 PM PDT, Sven-Haegar Koch wrote: >On Fri, 23 Sep 2016, Mike wrote: > >> On 09/23/2016 11:30 AM, Seth Mattinen wrote: >> > On 9/23/16 10:58, Grant Ridder wrote: >> > > Didn't realize Akamai kicked out or disabled customers >> > > >http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-after-ddos-attack-proves-pricey/ > >> > > >> > > "Security blog Krebs on Security has been taken offline by host >Akamai >> > > Technologies following a DDoS attack which reached 665 Gbps in >size." >> > >> > >> > So ultimately the DDoS was successful, just in a different way. >> > >> > ~Seth >> > >> > >> More technical information about the characteristics of these attacks >would be >> very interesting such as the ultimate sources of the attack traffic >> (compromised home pc's?), the nature of the traffic (dns / ssdp >> amplification?), whether it was spoofed source (BCP38-adverse), and >whether >> the recent takedown the vDOS was really complete or if it's likely >someone >> else gained control of the C&C servers that controlled it's assets? > >At least for the OVH case there is a bit of info: > >https://twitter.com/olesovhcom/status/779297257199964160 > >"This botnet with 145607 cameras/dvr (1-30Mbps per IP) is able to send >>1.5Tbps DDoS. Type: tcp/ack, tcp/ack+psh, tcp/syn." Krebs said it was mostly GRE. Pulling from the archive.org copy of his post[1]: "Preliminary analysis of the attack traffic suggests that perhaps the biggest chunk of the attack came in the form of traffic designed to look like it was generic routing encapsulation (GRE) data packets..." This bothered me, though: "McKeay explained that the source of GRE traffic can?t be spoofed or faked the same way DDoS attackers can spoof DNS traffic." Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet... >c'ya >sven-haegar > >-- >Three may keep a secret, if two of them are dead. >- Ben F. -- Hugo Slabbert?????? | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E?? | also on Signal [1] https://web.archive.org/web/20160922021000/http://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/ [2] http://blog.slabnet.com/post/gre-reflection/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 850 bytes Desc: not available URL: From jared at puck.nether.net Fri Sep 23 21:29:59 2016 From: jared at puck.nether.net (Jared Mauch) Date: Fri, 23 Sep 2016 17:29:59 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> Message-ID: <41F6A6E6-2545-40D3-8C01-380946E36C00@puck.nether.net> > On Sep 23, 2016, at 5:24 PM, Hugo Slabbert wrote: > > Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet... my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ?clean? traffic back to origin sites. - Jared From hugo at slabnet.com Fri Sep 23 21:39:54 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Fri, 23 Sep 2016 14:39:54 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <41F6A6E6-2545-40D3-8C01-380946E36C00@puck.nether.net> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> <41F6A6E6-2545-40D3-8C01-380946E36C00@puck.nether.net> Message-ID: <20160923213954.GC3770@bamboo.slabnet.com> On Fri 2016-Sep-23 17:29:59 -0400, Jared Mauch wrote: > >> On Sep 23, 2016, at 5:24 PM, Hugo Slabbert wrote: >> >> Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet... > >my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ?clean? traffic back to origin sites. But those tunnels generally wouldn't terminate on addresses within the protected block. If I'm protecting 192.0.22.0/24 for you, things would get confused if my GRE tunnel dest is somewhere in 192.0.22.0/24 because I would have a route for 192.0.22.0/24 across the GRE tunnel (either static or more conventionally learned via BGP). I'd bank it on either a provider touchdown or another unprotected block, otherwise I'd have a recursive routing issue where the tunnel destination would get routed through the GRE tunnel. I've been there with bouncy-bouncy tunnels on service turn-up when that was flubbed. Alternatively, I could pin /32s for the tunnel destinations and still learn the shorter protected block, but even so I should then know which source (my) and dest (customer's) IPs to exclude explicitly from the GRE filtering. If the attackers were hitting the GRE tunnel destination and spoofing the tunnel source that would make things harder, but that's starting to get into rather intimate knowledge of the scrubber's and customer's setup. I could still probably filter on e.g. TTLs or drop GRE further up to the northern edge on input rather than output, but agreed that is starting to get trickier... > >- Jared -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From mel at beckman.org Fri Sep 23 22:00:31 2016 From: mel at beckman.org (Mel Beckman) Date: Fri, 23 Sep 2016 22:00:31 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160923213954.GC3770@bamboo.slabnet.com> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> <41F6A6E6-2545-40D3-8C01-380946E36C00@puck.nether.net> <20160923213954.GC3770@bamboo.slabnet.com> Message-ID: <6E1AAFF1-F339-463F-896B-94ED754BE5B8@beckman.org> A similar GRE attack was used against the Olympics: "Once the Olympics got under way, LizardStresser along with a few other botnets ramped up their attack against organizations affiliated with the Olympics. The DDoS campaign launched attack traffic using the lesser-known IP protocol Generic Routing Encapsulation (GRE).? -mel On Sep 23, 2016, at 2:39 PM, Hugo Slabbert > wrote: On Fri 2016-Sep-23 17:29:59 -0400, Jared Mauch > wrote: On Sep 23, 2016, at 5:24 PM, Hugo Slabbert > wrote: Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet... my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ?clean? traffic back to origin sites. But those tunnels generally wouldn't terminate on addresses within the protected block. If I'm protecting 192.0.22.0/24 for you, things would get confused if my GRE tunnel dest is somewhere in 192.0.22.0/24 because I would have a route for 192.0.22.0/24 across the GRE tunnel (either static or more conventionally learned via BGP). I'd bank it on either a provider touchdown or another unprotected block, otherwise I'd have a recursive routing issue where the tunnel destination would get routed through the GRE tunnel. I've been there with bouncy-bouncy tunnels on service turn-up when that was flubbed. Alternatively, I could pin /32s for the tunnel destinations and still learn the shorter protected block, but even so I should then know which source (my) and dest (customer's) IPs to exclude explicitly from the GRE filtering. If the attackers were hitting the GRE tunnel destination and spoofing the tunnel source that would make things harder, but that's starting to get into rather intimate knowledge of the scrubber's and customer's setup. I could still probably filter on e.g. TTLs or drop GRE further up to the northern edge on input rather than output, but agreed that is starting to get trickier... - Jared -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal From deleskie at gmail.com Fri Sep 23 22:00:50 2016 From: deleskie at gmail.com (jim deleskie) Date: Fri, 23 Sep 2016 19:00:50 -0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: Not at all. I refered to AUP's as a way people remove you from a service when you use more of it then you are paying for. On Fri, Sep 23, 2016 at 3:58 PM, Marcin Cieslak wrote: > On Fri, 23 Sep 2016, jim deleskie wrote: > > > They were hosting him for free, and like insurance, I can assure you if > you > > are consistently using a service, and not covering the costs of that > > service you won't be a client for long. This is the basis for AUP/client > > contracts and have been going back to the days when we all offered only > > dialup internet. > > Does being a victim of a DDoS constitute a breach of AUP? > > Marcin Cie?lak > From jared at puck.nether.net Fri Sep 23 22:04:22 2016 From: jared at puck.nether.net (Jared Mauch) Date: Fri, 23 Sep 2016 18:04:22 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160923213954.GC3770@bamboo.slabnet.com> References: <0470bc20-12e9-528e-8465-52739f237c33@rollernet.us> <6c35f954-f05d-4b83-0250-44bc4270c083@tiedyenetworks.com> <71F8DB12-1EF0-4E0F-9F5A-59D3DC6F16A5@slabnet.com> <41F6A6E6-2545-40D3-8C01-380946E36C00@puck.nether.net> <20160923213954.GC3770@bamboo.slabnet.com> Message-ID: <7F85F9F8-9D92-4568-8C5D-AFB33EB60448@puck.nether.net> > On Sep 23, 2016, at 5:39 PM, Hugo Slabbert wrote: > > If the attackers were hitting the GRE tunnel destination and spoofing the tunnel source that would make things harder, but that's starting to get into rather intimate knowledge of the scrubber's and customer's setup. I could still probably filter on e.g. TTLs or drop GRE further up to the northern edge on input rather than output, but agreed that is starting to get trickier... My experiences are that under duress most people make poor choices and don?t properly filter these types of traffic. How many times have you turned off a filter to debug something? Making a tunnel work is trickier than it seems and not all devices can terminate them. In Cisco IOS land, you also have to have an Ip address on the tunnel for it to handle IP traffic, even if it?s ?ip unnumbered?. My guess is someone terminates on their P2P link to carrier, and that is easy enough to find w/ traceroute/mtr. - Jared From jlewis at lewis.org Sat Sep 24 01:24:58 2016 From: jlewis at lewis.org (Jon Lewis) Date: Fri, 23 Sep 2016 21:24:58 -0400 (EDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: On Fri, 23 Sep 2016, Patrick W. Gilmore wrote: > Is CloudFlare able to filter Layer 7 these days? I was under the > impression CloudFlare was not able to do that. > > There have been a lot of rumors about this attack. Some say reflection, > others say Layer 7, others say .. other stuff. If it is Layer 7, how are > you going to ??step in front of the cannon??? Would you just pass > through all the traffic? Anycast + load balancers + high powered varnish? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ From morrowc.lists at gmail.com Sat Sep 24 01:58:42 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Fri, 23 Sep 2016 21:58:42 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis wrote: > On Fri, 23 Sep 2016, Patrick W. Gilmore wrote: > > Is CloudFlare able to filter Layer 7 these days? I was under the >> impression CloudFlare was not able to do that. >> >> There have been a lot of rumors about this attack. Some say reflection, >> others say Layer 7, others say .. other stuff. If it is Layer 7, how are >> you going to ??step in front of the cannon??? Would you just pass through >> all the traffic? >> > > Anycast + load balancers + high powered varnish? > > notionally (because I have been paying zero attention to this) jon's suggesting: 1) setup a crapload of nginx/squid/etc configured tightly for things to be accessed behind them 2) ecmp to them across several layers (assume 32 ecmp at each layer, call it 4 layers get craploads of machines running) 3) change over the dns 4) profit-- eh? If you can eat the PPS, you can spray across enough tcp listeners, you can weed out the chaff and start filtering in the 'application'... perhaps also run a 'low bandwidth' version of the target site... hey look, we invented prolexic. From jlewis at lewis.org Sat Sep 24 02:13:45 2016 From: jlewis at lewis.org (Jon Lewis) Date: Fri, 23 Sep 2016 22:13:45 -0400 (EDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: On Fri, 23 Sep 2016, Christopher Morrow wrote: > On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis wrote: > >> On Fri, 23 Sep 2016, Patrick W. Gilmore wrote: >> >> Is CloudFlare able to filter Layer 7 these days? I was under the >>> impression CloudFlare was not able to do that. >>> >>> There have been a lot of rumors about this attack. Some say reflection, >>> others say Layer 7, others say .. other stuff. If it is Layer 7, how are >>> you going to ??step in front of the cannon??? Would you just pass through >>> all the traffic? >>> >> >> Anycast + load balancers + high powered varnish? >> >> > notionally (because I have been paying zero attention to this) jon's > suggesting: > 1) setup a crapload of nginx/squid/etc configured tightly for things to > be accessed behind them > 2) ecmp to them across several layers (assume 32 ecmp at each layer, call > it 4 layers get craploads of machines running) > 3) change over the dns > 4) profit-- > > eh? If you can eat the PPS, you can spray across enough tcp listeners, you > can weed out the chaff and start filtering in the 'application'... perhaps > also run a 'low bandwidth' version of the target site... > > hey look, we invented prolexic. Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around 50G per site, and at that level, filtering the obvious crap gets much more reasonable. Then, doing the layer 7 scrubbing of the less obvious crap is more easily dealt with than a single site receiving 600G of attack traffic. I haven't actually done this (specifically for DDoS mitigation)...just speculating as to how it might easily be done given sufficient resources. The trouble is, the attackers have virtually unlimited bandwidth, and aren't constrained by having to pay for the bandwidth. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ From morrowc.lists at gmail.com Sat Sep 24 02:42:45 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Fri, 23 Sep 2016 22:42:45 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <2042806831.1292.1474653708642.JavaMail.mhammett@ThunderFuck> Message-ID: On Fri, Sep 23, 2016 at 10:13 PM, Jon Lewis wrote: > On Fri, 23 Sep 2016, Christopher Morrow wrote: > > On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis wrote: >> >> On Fri, 23 Sep 2016, Patrick W. Gilmore wrote: >>> >>> Is CloudFlare able to filter Layer 7 these days? I was under the >>> >>>> impression CloudFlare was not able to do that. >>>> >>>> There have been a lot of rumors about this attack. Some say reflection, >>>> others say Layer 7, others say .. other stuff. If it is Layer 7, how are >>>> you going to ??step in front of the cannon??? Would you just pass >>>> through >>>> all the traffic? >>>> >>>> >>> Anycast + load balancers + high powered varnish? >>> >>> >>> notionally (because I have been paying zero attention to this) jon's >> suggesting: >> 1) setup a crapload of nginx/squid/etc configured tightly for things to >> be accessed behind them >> 2) ecmp to them across several layers (assume 32 ecmp at each layer, call >> it 4 layers get craploads of machines running) >> 3) change over the dns >> 4) profit-- >> >> eh? If you can eat the PPS, you can spray across enough tcp listeners, you >> can weed out the chaff and start filtering in the 'application'... perhaps >> also run a 'low bandwidth' version of the target site... >> >> hey look, we invented prolexic. >> > > Well...by anycast, I meant BGP anycast, spreading the "target" > geographically to a dozen or more well connected/peered origins. At that > point, your ~600G DDoS might only be around anycast and tcp? the heck you say! :) > 50G per site, and at that level, filtering the obvious crap gets much more > reasonable. Then, doing the layer 7 scrubbing of the less obvious crap is > more easily dealt with than a single site receiving 600G of attack traffic. > > sure, yes. > I haven't actually done this (specifically for DDoS mitigation)...just > speculating as to how it might easily be done given sufficient resources. > The trouble is, the attackers have virtually unlimited bandwidth, and > aren't constrained by having to pay for the bandwidth. > > sounds like you got it all sorted out... > > ---------------------------------------------------------------------- > Jon Lewis, MCP :) | I route > | therefore you are > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ > From johnl at iecc.com Sat Sep 24 14:47:57 2016 From: johnl at iecc.com (John Levine) Date: 24 Sep 2016 14:47:57 -0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Message-ID: <20160924144757.6291.qmail@ary.lan> >> Well...by anycast, I meant BGP anycast, spreading the "target" >> geographically to a dozen or more well connected/peered origins. At that >> point, your ~600G DDoS might only be around > >anycast and tcp? the heck you say! :) People who've tried it say it works fine. Routes don't flap that often. From woody at pch.net Sat Sep 24 16:28:24 2016 From: woody at pch.net (Bill Woodcock) Date: Sat, 24 Sep 2016 09:28:24 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160924144757.6291.qmail@ary.lan> References: <20160924144757.6291.qmail@ary.lan> Message-ID: <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> > On Sep 24, 2016, at 7:47 AM, John Levine wrote: > >>> Well...by anycast, I meant BGP anycast, spreading the "target" >>> geographically to a dozen or more well connected/peered origins. At that >>> point, your ~600G DDoS might only be around >> >> anycast and tcp? the heck you say! :) > > People who've tried it say it works fine. It?s worked fine for 28 years, for me. -Bill -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From morrowc.lists at gmail.com Sat Sep 24 16:55:22 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Sat, 24 Sep 2016 12:55:22 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> Message-ID: On Sat, Sep 24, 2016 at 12:28 PM, Bill Woodcock wrote: > > > On Sep 24, 2016, at 7:47 AM, John Levine wrote: > > > >>> Well...by anycast, I meant BGP anycast, spreading the "target" > >>> geographically to a dozen or more well connected/peered origins. At > that > >>> point, your ~600G DDoS might only be around > >> > >> anycast and tcp? the heck you say! :) > > > > People who've tried it say it works fine. > > It?s worked fine for 28 years, for me. > > > boy, it'd sure be nice if there were some 'science' and 'measurement' behind such statements. Didn't k-root do some anycast studies ~8-10 years back? -chris (note I'm totally a believer in anycast for tcp in the 'right' circumstances, but often it feels like talking to climate-change-deniers when proffering it as a solution) From niels=nanog at bakker.net Sat Sep 24 18:43:32 2016 From: niels=nanog at bakker.net (Niels Bakker) Date: Sat, 24 Sep 2016 20:43:32 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> Message-ID: <20160924184332.GA45065@excession.tpb.net> * morrowc.lists at gmail.com (Christopher Morrow) [Sat 24 Sep 2016, 18:55 CEST]: >boy, it'd sure be nice if there were some 'science' and >'measurement' behind such statements. >Didn't k-root do some anycast studies ~8-10 years back? Not k-root but CacheFly 2006: https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf -- Niels. From morrowc.lists at gmail.com Sat Sep 24 18:53:32 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Sat, 24 Sep 2016 14:53:32 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160924184332.GA45065@excession.tpb.net> References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> Message-ID: On Sat, Sep 24, 2016 at 2:43 PM, Niels Bakker wrote: > * morrowc.lists at gmail.com (Christopher Morrow) [Sat 24 Sep 2016, 18:55 > CEST]: > >> boy, it'd sure be nice if there were some 'science' and 'measurement' >> behind such statements. >> Didn't k-root do some anycast studies ~8-10 years back? >> > > Not k-root but CacheFly 2006: https://www.nanog.org/meetings > /nanog37/presentations/matt.levine.pdf > > > that's not the one I was thinking of, this is: which references your presentation, nice! and is about J-root, not K-root, but mentions Lorenzo's work on K-root studies... In anycase, both seem to say that 'tcp anycast works fine' (inside some set of parameters). thanks! -chris From brett at the-watsons.org Sat Sep 24 21:16:05 2016 From: brett at the-watsons.org (Brett Watson) Date: Sat, 24 Sep 2016 14:16:05 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> Message-ID: >> > that's not the one I was thinking of, this is: > > > which references your presentation, nice! and is about J-root, not K-root, > but mentions Lorenzo's work on K-root studies... In anycase, both seem to > say that 'tcp anycast works fine' (inside some set of parameters). > Right? and we?ve known this since about? ? 1996? From jra at baylink.com Sun Sep 25 00:54:33 2016 From: jra at baylink.com (Jay R. Ashworth) Date: Sun, 25 Sep 2016 00:54:33 +0000 (UTC) Subject: One Year On: IPv4 Exhaust Message-ID: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John Curran announced that the last IPv4 address block in ARIN's Free Pool had been assigned. How's that been workin' out for everyone? Cheers, -- jra -- Jay R. Ashworth Baylink jra at baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 From justin at cloudflare.com Sun Sep 25 01:28:28 2016 From: justin at cloudflare.com (Justin Paine) Date: Sun, 25 Sep 2016 01:28:28 +0000 (UTC) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> Message-ID: <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45 On Google now.? ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 711557B6 0114 DE0B 314D On Sat, Sep 24, 2016 at 2:17 PM -0700, "Brett Watson" wrote: >> > that's not the one I was thinking of, this is: > > > which references your presentation, nice! and is about J-root, not K-root, > but mentions Lorenzo's work on K-root studies... In anycase, both seem to > say that 'tcp anycast works fine' (inside some set of parameters). > Right? and we?ve known this since about? ? 1996? From jared at puck.nether.net Sun Sep 25 01:51:55 2016 From: jared at puck.nether.net (Jared Mauch) Date: Sat, 24 Sep 2016 21:51:55 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> Message-ID: > On Sep 24, 2016, at 9:28 PM, Justin Paine via NANOG wrote: > > > DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45 I recommend running this command (or similar): rndc flushname krebsonsecurity.com if you still see 127.0.0.1 - Jared From jayfar at jayfar.com Sun Sep 25 02:45:13 2016 From: jayfar at jayfar.com (Jay Farrell) Date: Sat, 24 Sep 2016 22:45:13 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> Message-ID: And of course on windows ipconfig /flushdns Still I had to wait for my corporate caching servers to update; I think the TTL on the old A record was an hour. On Sat, Sep 24, 2016 at 9:51 PM, Jared Mauch wrote: > > > On Sep 24, 2016, at 9:28 PM, Justin Paine via NANOG > wrote: > > > > > > DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com > 157 IN A 130.211.45.45 > > > I recommend running this command (or similar): > > rndc flushname krebsonsecurity.com > > if you still see 127.0.0.1 > > - Jared From cb.list6 at gmail.com Sun Sep 25 02:46:59 2016 From: cb.list6 at gmail.com (Ca By) Date: Sat, 24 Sep 2016 19:46:59 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> References: <20160924144757.6291.qmail@ary.lan> <3CF1B7DD-BAE0-494A-B972-7E153A22E449@pch.net> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> Message-ID: On Saturday, September 24, 2016, Justin Paine via NANOG wrote: > > DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 > IN A 130.211.45.45 > > On Google now. > > Next question. Will google use the information from the telemetry, rumored to be webcams, to defang the bot ? Like post an informative message that the source network is hosting a bad actor (same nat ipv4, /25, ... ). Or , work with the access networks (Comcast, rcs/rds, china telecom) to disconnect and manage the abusers ? > From jra at baylink.com Sun Sep 25 04:43:08 2016 From: jra at baylink.com (Jay R. Ashworth) Date: Sun, 25 Sep 2016 04:43:08 +0000 (UTC) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> Message-ID: <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> ----- Original Message ----- > From: "Jay Farrell via NANOG" > And of course on windows ipconfig /flushdns > > Still I had to wait for my corporate caching servers to update; I think the > TTL on the old A record was an hour. Are big eyeball networks still flooring A record TTLs on resolution? Cheers, -- jra -- Jay R. Ashworth Baylink jra at baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 From henson at acm.org Sun Sep 25 07:30:16 2016 From: henson at acm.org (Paul B. Henson) Date: Sun, 25 Sep 2016 00:30:16 -0700 Subject: Status of IPv6 on Charter Communications In-Reply-To: <20160910151413.so36rwjawtuhlvai@560338596deac36a23b4268bcf651d8275290d1da332da34> References: <20160910151413.so36rwjawtuhlvai@560338596deac36a23b4268bcf651d8275290d1da332da34> Message-ID: <20160925073015.GB4221@bender.unx.cpp.edu> On Sat, Sep 10, 2016 at 11:14:13AM -0400, David Hill wrote: > On Sat, Sep 10, 2016 at 06:55:59AM -0700, Stephen Satchell wrote: > > Would someone at Charter Communications who is on this list indicate the > > roll-out schedule for IPv6 to business customers using cable modems as > > opposed to fiber links? > > I too would appreciate this information. I do see more networks being > announced: http://bgp.he.net/AS20115#_prefixes6. Hopefully they will be > providing static IPv6 as well. Would somebody at Frontier FIOS who's embarassed about everyone else dropping out IPv6 before them and would like to show up Verizon who so totally dropped the ball on that make some kind, *any* kind, of an announcement on even a vague potential timeline on such a roll-out schedule too ;)? Still waiting to be contacted about my static IP cutover here in CA. Almost halfway through the transition year of the loaner Verizon IP space... Anybody with a business static gone through the migration yet? I'm planning to ask them about IPv6 then, but I'm sure I'll get the same "we don't know anything" I got from Verizon for 4 years and from Frontier when I asked right after the cutover . From jayfar at jayfar.com Sun Sep 25 13:22:40 2016 From: jayfar at jayfar.com (Jay Farrell) Date: Sun, 25 Sep 2016 09:22:40 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> Message-ID: And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?). https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth wrote: > ----- Original Message ----- > > From: "Jay Farrell via NANOG" > > > And of course on windows ipconfig /flushdns > > > > Still I had to wait for my corporate caching servers to update; I think > the > > TTL on the old A record was an hour. > > Are big eyeball networks still flooring A record TTLs on resolution? > > Cheers, > -- jra > -- > Jay R. Ashworth Baylink > jra at baylink.com > Designer The Things I Think RFC > 2100 > Ashworth & Associates http://www.bcp38.info 2000 Land > Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > 1274 > From jra at baylink.com Sun Sep 25 14:32:11 2016 From: jra at baylink.com (Jay R. Ashworth) Date: Sun, 25 Sep 2016 14:32:11 +0000 (UTC) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> Message-ID: <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> ----- Original Message ----- > From: "Jay Farrell via NANOG" > And of course Brian Krebs has a thing or two to say, not the least is which > to push for BCP38 (good luck with that, right?). > > https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ Well, given how few contributions we've gotten at bcp38.info in the last, what, 4 years, yeah, I guess so... Cheers, -- jra -- Jay R. Ashworth Baylink jra at baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 From cb.list6 at gmail.com Sun Sep 25 14:36:18 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 07:36:18 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> Message-ID: On Sunday, September 25, 2016, Jay Farrell via NANOG wrote: > And of course Brian Krebs has a thing or two to say, not the least is which > to push for BCP38 (good luck with that, right?). > > https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > Yeh, bcp38 is not a viable solution. As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle. A solution is aggregating the telemetry of source IP addresses in the botnet and assigning blame and liability to the owners of the IP addresses / host ASN. The networks can then use AUP to shutdown the bot members. As where http://openntpproject.org/ was a proactive approach, Kreb's data can be reactive approach. And since the data is evidence of a crime, the network operators can enforce the AUP. The attack did happen. This ip was involved. Remediation is required. >From there, the host ASN can > On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth > wrote: > > > ----- Original Message ----- > > > From: "Jay Farrell via NANOG" > > > > > > And of course on windows ipconfig /flushdns > > > > > > Still I had to wait for my corporate caching servers to update; I think > > the > > > TTL on the old A record was an hour. > > > > Are big eyeball networks still flooring A record TTLs on resolution? > > > > Cheers, > > -- jra > > -- > > Jay R. Ashworth Baylink > > jra at baylink.com > > Designer The Things I Think RFC > > 2100 > > Ashworth & Associates http://www.bcp38.info 2000 Land > > Rover DII > > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > > 1274 > > > From nanog at ics-il.net Sun Sep 25 14:50:54 2016 From: nanog at ics-il.net (Mike Hammett) Date: Sun, 25 Sep 2016 09:50:54 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> Message-ID: <170256573.728.1474815053840.JavaMail.mhammett@ThunderFuck> I've heard people say doing BCP38 is hard for big networks and it is if you do it at your provider\peering edges. It's easier if done at the customer edge. Simply don't allow the traffic onto your network to start with. Limit the spoofing attacks to just a single random ASN. How much smaller is the attack than it is now with hundreds or thousands of them? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Ca By" To: "Jay Farrell" Cc: "North American Network Operators' Group" Sent: Sunday, September 25, 2016 9:36:18 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sunday, September 25, 2016, Jay Farrell via NANOG wrote: > And of course Brian Krebs has a thing or two to say, not the least is which > to push for BCP38 (good luck with that, right?). > > https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > Yeh, bcp38 is not a viable solution. As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle. A solution is aggregating the telemetry of source IP addresses in the botnet and assigning blame and liability to the owners of the IP addresses / host ASN. The networks can then use AUP to shutdown the bot members. As where http://openntpproject.org/ was a proactive approach, Kreb's data can be reactive approach. And since the data is evidence of a crime, the network operators can enforce the AUP. The attack did happen. This ip was involved. Remediation is required. >From there, the host ASN can > On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth > wrote: > > > ----- Original Message ----- > > > From: "Jay Farrell via NANOG" > > > > > > And of course on windows ipconfig /flushdns > > > > > > Still I had to wait for my corporate caching servers to update; I think > > the > > > TTL on the old A record was an hour. > > > > Are big eyeball networks still flooring A record TTLs on resolution? > > > > Cheers, > > -- jra > > -- > > Jay R. Ashworth Baylink > > jra at baylink.com > > Designer The Things I Think RFC > > 2100 > > Ashworth & Associates http://www.bcp38.info 2000 Land > > Rover DII > > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > > 1274 > > > From jra at baylink.com Sun Sep 25 15:01:44 2016 From: jra at baylink.com (Jay R. Ashworth) Date: Sun, 25 Sep 2016 15:01:44 +0000 (UTC) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> Message-ID: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> ----- Original Message ----- > From: "Ca By" > On Sunday, September 25, 2016, Jay Farrell via NANOG > wrote: > >> And of course Brian Krebs has a thing or two to say, not the least is which >> to push for BCP38 (good luck with that, right?). >> >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > Yeh, bcp38 is not a viable solution. > > As long as their is one spoof capable network on the net, the problem will > not be solved. While bcp38 is a true bcp, it is not a solution. It will > not, and has not, moved the needle. No; things which are not implemented anywhere generally don't move the needle. You're confusing cause and effect here, I think. You give no evidence that *pervasive implementation of 38* would *not* move the needle, and that's where we are right now: we do not have anything that looks like "pervasive implementation". *Ten* people could solve this problem. Tomorrow. The chief engineers of the top 10 US eyeball providers could simply sit down and say "let's go do this thing". And better than 80% of the potential sources would just vanish off the face of the internet. Do I need to go do research, and name these 10 people? :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra at baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 From cb.list6 at gmail.com Sun Sep 25 15:13:24 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 08:13:24 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> Message-ID: On Sunday, September 25, 2016, Jay R. Ashworth wrote: > ----- Original Message ----- > > From: "Ca By" > > > > On Sunday, September 25, 2016, Jay Farrell via NANOG > > > wrote: > > > >> And of course Brian Krebs has a thing or two to say, not the least is > which > >> to push for BCP38 (good luck with that, right?). > >> > >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > > > Yeh, bcp38 is not a viable solution. > > > > As long as their is one spoof capable network on the net, the problem > will > > not be solved. While bcp38 is a true bcp, it is not a solution. It will > > not, and has not, moved the needle. > > No; things which are not implemented anywhere generally don't move the > needle. > > It is implemented many places in fact. > You're confusing cause and effect here, I think. > > I will argue you are confused. > You give no evidence that *pervasive implementation of 38* would *not* move > the needle, and that's where we are right now: we do not have anything that > looks like "pervasive implementation". > > *Ten* people could solve this problem. Tomorrow. > > The chief engineers of the top 10 US eyeball providers could simply sit > down > and say "let's go do this thing". And better than 80% of the potential > sources > would just vanish off the face of the internet. > > Assume every network in the usa implements bcp38. This simply means no spoofs source from usa. Every packet is sent from the usa using a valid origin. Assume also 50% of networks in Europe and Asia and the Southern Hemisphere do bcp38 too. Great. The result is the needle has not moved at all. CC nodes in the non bcp38 locations will send spoofed packets destinations is comcast and att with a source of krebs. Result? Comcast and att cpe responds with crap to krebs. Ddos success despite bcp38 in all of usa. > Do I need to go do research, and name these 10 people? :-) > > Cheers, > -- jra > -- > Jay R. Ashworth Baylink > jra at baylink.com > Designer The Things I Think RFC > 2100 > Ashworth & Associates http://www.bcp38.info 2000 Land > Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > 1274 > From paul at prt.org Sun Sep 25 16:19:01 2016 From: paul at prt.org (Paul Thornton) Date: Sun, 25 Sep 2016 17:19:01 +0100 Subject: One Year On: IPv4 Exhaust In-Reply-To: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> Message-ID: <57E7F8F5.6070402@prt.org> On 25/09/2016 01:54, Jay R. Ashworth wrote: > One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John > Curran announced that the last IPv4 address block in ARIN's Free Pool had > been assigned. > > How's that been workin' out for everyone? If you'll all indulge a bit of a RIPE-centric reply on this; I've was allocated a /22 from around half-way through 185.169.0.0/16 last week (185 being RIPE's final /8). Assuming that RIPE are allocating sequentially - and I believe they are - This means that they have consumed around 66.5% of their final /8. They started allocating from this in September 2012, which suggests a reasonably low consumption rate but the RIPE final /8 will be exhausted in around two years time. I can't find an equivalent ARIN page of "how much we've allocated from our last /8" - the statistics show that just over 2x /16s worth have been assigned/allocated between January 2016 and July 2016, so a lower rate by some margin than RIPE - but there are of course policy differences at play there. Now the operational question of "How has this affected us" is probably best answered with "We've had to pay real money for IPv4 addresses since then." What may be much more interesting is what happens when the fairly ready supply of IPv4 addresses in the secondary transfer market starts to dry up. Just throwing additional money at the problem will probably not be an effective or viable solution then. I'm sure that Geoff Huston has a much more accurate and colourful set of predictions than my back-of-envelope calculations for those interested! Paul. From cb.list6 at gmail.com Sun Sep 25 16:29:02 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 09:29:02 -0700 Subject: One Year On: IPv4 Exhaust In-Reply-To: <57E7F8F5.6070402@prt.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: On Sunday, September 25, 2016, Paul Thornton wrote: > On 25/09/2016 01:54, Jay R. Ashworth wrote: > >> One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John >> Curran announced that the last IPv4 address block in ARIN's Free Pool had >> been assigned. >> >> How's that been workin' out for everyone? >> > > If you'll all indulge a bit of a RIPE-centric reply on this; I've was > allocated a /22 from around half-way through 185.169.0.0/16 last week > (185 being RIPE's final /8). > > Assuming that RIPE are allocating sequentially - and I believe they are - > This means that they have consumed around 66.5% of their final /8. They > started allocating from this in September 2012, which suggests a reasonably > low consumption rate but the RIPE final /8 will be exhausted in around two > years time. > > I can't find an equivalent ARIN page of "how much we've allocated from our > last /8" - the statistics show that just over 2x /16s worth have been > assigned/allocated between January 2016 and July 2016, so a lower rate by > some margin than RIPE - but there are of course policy differences at play > there. > > Now the operational question of "How has this affected us" is probably > best answered with "We've had to pay real money for IPv4 addresses since > then." What may be much more interesting is what happens when the fairly > ready supply of IPv4 addresses in the secondary transfer market starts to > dry up. Just throwing additional money at the problem will probably not be > an effective or viable solution then. > > I'm sure that Geoff Huston has a much more accurate and colourful set of > predictions than my back-of-envelope calculations for those interested! > > Paul. > For your use case , would ipv6 solve anything? Think it is fair to say big content and big eyeballs have moved to IPv6 (notable exceptions exist) http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ From paul at prt.org Sun Sep 25 16:38:17 2016 From: paul at prt.org (Paul Thornton) Date: Sun, 25 Sep 2016 17:38:17 +0100 Subject: One Year On: IPv4 Exhaust In-Reply-To: References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: <57E7FD79.9000201@prt.org> On 25/09/2016 17:29, Ca By wrote: > For your use case , would ipv6 solve anything? > > Think it is fair to say big content and big eyeballs have moved to IPv6 > (notable exceptions exist) > > http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ Yes of course. Let's make the assumption that these people are happily v6 enabled but need to support v4 for the foreseeable future. Take, for example, large hosting environments. NAT isn't an option, nor is v6 only at this point. For them, the only option to provide unique v4 addresses for customers is to purchase it. We may be in luck, and the v6 tipping point happens before the transfer market runs out of reasonably-priced supply, and our hypothetical example above can default to v6 only. If that happens, fantastic - but I'm not sure I'd bet on it, even given the improved v6 takeup in the past year or two. Paul. From nanog at ics-il.net Sun Sep 25 16:57:00 2016 From: nanog at ics-il.net (Mike Hammett) Date: Sun, 25 Sep 2016 11:57:00 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> Message-ID: <725453796.753.1474822617423.JavaMail.mhammett@ThunderFuck> You don't need complete adoption to reduce the attacks. If ASes representing 25% of the current spoofed traffic implemented BCP38, then guess what, there's 25% less of an attack. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Ca By" To: "Jay R. Ashworth" Cc: "North American Network Operators' Group" Sent: Sunday, September 25, 2016 10:13:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sunday, September 25, 2016, Jay R. Ashworth wrote: > ----- Original Message ----- > > From: "Ca By" > > > > On Sunday, September 25, 2016, Jay Farrell via NANOG > > > wrote: > > > >> And of course Brian Krebs has a thing or two to say, not the least is > which > >> to push for BCP38 (good luck with that, right?). > >> > >> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > > > > Yeh, bcp38 is not a viable solution. > > > > As long as their is one spoof capable network on the net, the problem > will > > not be solved. While bcp38 is a true bcp, it is not a solution. It will > > not, and has not, moved the needle. > > No; things which are not implemented anywhere generally don't move the > needle. > > It is implemented many places in fact. > You're confusing cause and effect here, I think. > > I will argue you are confused. > You give no evidence that *pervasive implementation of 38* would *not* move > the needle, and that's where we are right now: we do not have anything that > looks like "pervasive implementation". > > *Ten* people could solve this problem. Tomorrow. > > The chief engineers of the top 10 US eyeball providers could simply sit > down > and say "let's go do this thing". And better than 80% of the potential > sources > would just vanish off the face of the internet. > > Assume every network in the usa implements bcp38. This simply means no spoofs source from usa. Every packet is sent from the usa using a valid origin. Assume also 50% of networks in Europe and Asia and the Southern Hemisphere do bcp38 too. Great. The result is the needle has not moved at all. CC nodes in the non bcp38 locations will send spoofed packets destinations is comcast and att with a source of krebs. Result? Comcast and att cpe responds with crap to krebs. Ddos success despite bcp38 in all of usa. > Do I need to go do research, and name these 10 people? :-) > > Cheers, > -- jra > -- > Jay R. Ashworth Baylink > jra at baylink.com > Designer The Things I Think RFC > 2100 > Ashworth & Associates http://www.bcp38.info 2000 Land > Rover DII > St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 > 1274 > From jtk at depaul.edu Sun Sep 25 17:00:21 2016 From: jtk at depaul.edu (John Kristoff) Date: Sun, 25 Sep 2016 12:00:21 -0500 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> Message-ID: <20160925120021.79280a95@p50.localdomain> On Sun, 25 Sep 2016 14:36:18 +0000 Ca By wrote: > As long as their is one spoof capable network on the net, the problem will > not be solved. This is not strictly true. If it could be determined where a large bulk of the spoofing came from, public pressure could be applied. This may not have been the issue in this case, but in many amplification and reflection attacks, the originating spoof-enabled networks were from a limited set of networks. De-peering, service termination, shaming, etc could have an effect. John From nanog at ics-il.net Sun Sep 25 17:12:06 2016 From: nanog at ics-il.net (Mike Hammett) Date: Sun, 25 Sep 2016 12:12:06 -0500 (CDT) Subject: One Year On: IPv4 Exhaust In-Reply-To: <57E7F8F5.6070402@prt.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: <1011263719.788.1474823524301.JavaMail.mhammett@ThunderFuck> ARIN exhausted their last /8 about a year ago. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Paul Thornton" To: nanog at nanog.org Sent: Sunday, September 25, 2016 11:19:01 AM Subject: Re: One Year On: IPv4 Exhaust On 25/09/2016 01:54, Jay R. Ashworth wrote: > One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John > Curran announced that the last IPv4 address block in ARIN's Free Pool had > been assigned. > > How's that been workin' out for everyone? If you'll all indulge a bit of a RIPE-centric reply on this; I've was allocated a /22 from around half-way through 185.169.0.0/16 last week (185 being RIPE's final /8). Assuming that RIPE are allocating sequentially - and I believe they are - This means that they have consumed around 66.5% of their final /8. They started allocating from this in September 2012, which suggests a reasonably low consumption rate but the RIPE final /8 will be exhausted in around two years time. I can't find an equivalent ARIN page of "how much we've allocated from our last /8" - the statistics show that just over 2x /16s worth have been assigned/allocated between January 2016 and July 2016, so a lower rate by some margin than RIPE - but there are of course policy differences at play there. Now the operational question of "How has this affected us" is probably best answered with "We've had to pay real money for IPv4 addresses since then." What may be much more interesting is what happens when the fairly ready supply of IPv4 addresses in the secondary transfer market starts to dry up. Just throwing additional money at the problem will probably not be an effective or viable solution then. I'm sure that Geoff Huston has a much more accurate and colourful set of predictions than my back-of-envelope calculations for those interested! Paul. From johnl at iecc.com Sun Sep 25 17:26:09 2016 From: johnl at iecc.com (John Levine) Date: 25 Sep 2016 17:26:09 -0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> Message-ID: <20160925172609.10758.qmail@ary.lan> >> Yeh, bcp38 is not a viable solution. Krebs said this DDoS came from insecure IoT devices, of which there are a kazillion, with the numbers growing every day. Why would they need to spoof IPs? How would BCP38 help? R's, John From cb.list6 at gmail.com Sun Sep 25 17:27:01 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 10:27:01 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160925120021.79280a95@p50.localdomain> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: On Sunday, September 25, 2016, John Kristoff wrote: > On Sun, 25 Sep 2016 14:36:18 +0000 > Ca By > wrote: > > > As long as their is one spoof capable network on the net, the problem > will > > not be solved. > > This is not strictly true. If it could be determined where a large > bulk of the spoofing came from, public pressure could be applied. This > may not have been the issue in this case, but in many amplification and > reflection attacks, the originating spoof-enabled networks were from a > limited set of networks. De-peering, service termination, shaming, etc > could have an effect. > > John > Ok, sorry for the not being exact. I am trying to be practical. My point is, a lot of access networks will respond to public pressure if the data is exposed on the offending real ips of the iot crap, and they will enforce their AUP. We have seen comcast do just that, on this list a few months back. That path has legs. Google also blocks service to certain hacked networks as well, we have seen that on this list too. That is an interesting angle in the krebs case. Will google block service to folks sharing ip with the iot ddos mess ? From cb.list6 at gmail.com Sun Sep 25 17:34:33 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 10:34:33 -0700 Subject: One Year On: IPv4 Exhaust In-Reply-To: <57E7FD79.9000201@prt.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <57E7FD79.9000201@prt.org> Message-ID: On Sunday, September 25, 2016, Paul Thornton wrote: > > On 25/09/2016 17:29, Ca By wrote: > > For your use case , would ipv6 solve anything? >> >> Think it is fair to say big content and big eyeballs have moved to IPv6 >> (notable exceptions exist) >> >> http://www.internetsociety.org/deploy360/blog/2016/08/facebo >> ok-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ >> > > Yes of course. Let's make the assumption that these people are happily v6 > enabled but need to support v4 for the foreseeable future. > > Take, for example, large hosting environments. NAT isn't an option, nor > is v6 only at this point. For them, the only option to provide unique v4 > addresses for customers is to purchase it. > > We may be in luck, and the v6 tipping point happens before the transfer > market runs out of reasonably-priced supply, and our hypothetical example > above can default to v6 only. If that happens, fantastic - but I'm not > sure I'd bet on it, even given the improved v6 takeup in the past year or > two. > > Paul. > I think how this will work out is that IPv4 becomes decoupled from hosting / cloud, and those IPv4 service have to be shared via L7 load balancing and / or CDN that has ipv4. Meaning hosts have ipv6 and need to subscribe to "ipv4 as a service " I think the big networks are sharding based on ip protocol. Here is stack for ipv4 (decling use), here is a stack for ipv6 (increasing use, over 50% of all traffic in many cases today, especially mobile) The idea of dual stack probably wont last long. The service is available as dual stack, but the back end is real ipv6 and magic hack ipv4. Just $0.02 on trajectory From sethm at rollernet.us Sun Sep 25 17:40:41 2016 From: sethm at rollernet.us (Seth Mattinen) Date: Sun, 25 Sep 2016 10:40:41 -0700 Subject: One Year On: IPv4 Exhaust In-Reply-To: <57E7F8F5.6070402@prt.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: <5cc20723-80da-0b21-4f46-86b730223114@rollernet.us> On 9/25/16 9:19 AM, Paul Thornton wrote: > > I can't find an equivalent ARIN page of "how much we've allocated from > our last /8" - the statistics show that just over 2x /16s worth have > been assigned/allocated between January 2016 and July 2016, so a lower > rate by some margin than RIPE - but there are of course policy > differences at play there. ARIN's last /8 was run to zero last year. Anything since then has been randomness from the waiting list such as: https://www.arin.net/announcements/2016/20160902.html ~Seth From cb.list6 at gmail.com Sun Sep 25 17:41:12 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 10:41:12 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160925172609.10758.qmail@ary.lan> References: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> <20160925172609.10758.qmail@ary.lan> Message-ID: On Sunday, September 25, 2016, John Levine wrote: > >> Yeh, bcp38 is not a viable solution. > > Krebs said this DDoS came from insecure IoT devices, of which there > are a kazillion, with the numbers growing every day. Why would they > need to spoof IPs? How would BCP38 help? > > R's, > John > Worth reading to level set https://www.internetsociety.org/sites/default/files/01_5.pdf The attack is triggered by a few spoofs somewhere in the world. It is not feasible to stop this. The attack traffic that blows up to 600gbs is from traceable iot crap , the victim knows who is sending the packers (iot crap) and the access network (comcast, att ...) has the AUP authority to shut it down. One by one. Or automated. Please see https://www.ietf.org/rfc/rfc6561.txt From deleskie at gmail.com Sun Sep 25 17:41:21 2016 From: deleskie at gmail.com (jim deleskie) Date: Sun, 25 Sep 2016 14:41:21 -0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and medium sized networks. What does it cost and what do we make doing it, over rules what is "good for the internet" every time it came up. On Sun, Sep 25, 2016 at 2:27 PM, Ca By wrote: > On Sunday, September 25, 2016, John Kristoff wrote: > > > On Sun, 25 Sep 2016 14:36:18 +0000 > > Ca By > wrote: > > > > > As long as their is one spoof capable network on the net, the problem > > will > > > not be solved. > > > > This is not strictly true. If it could be determined where a large > > bulk of the spoofing came from, public pressure could be applied. This > > may not have been the issue in this case, but in many amplification and > > reflection attacks, the originating spoof-enabled networks were from a > > limited set of networks. De-peering, service termination, shaming, etc > > could have an effect. > > > > John > > > > Ok, sorry for the not being exact. I am trying to be practical. > > My point is, a lot of access networks will respond to public pressure if > the data is exposed on the offending real ips of the iot crap, and they > will enforce their AUP. > > We have seen comcast do just that, on this list a few months back. That > path has legs. > > Google also blocks service to certain hacked networks as well, we have seen > that on this list too. That is an interesting angle in the krebs case. Will > google block service to folks sharing ip with the iot ddos mess ? > From paul at prt.org Sun Sep 25 17:47:54 2016 From: paul at prt.org (Paul Thornton) Date: Sun, 25 Sep 2016 18:47:54 +0100 Subject: One Year On: IPv4 Exhaust In-Reply-To: <5cc20723-80da-0b21-4f46-86b730223114@rollernet.us> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <5cc20723-80da-0b21-4f46-86b730223114@rollernet.us> Message-ID: <57E80DCA.8010205@prt.org> On 25/09/2016 18:40, Seth Mattinen wrote: > On 9/25/16 9:19 AM, Paul Thornton wrote: >> >> I can't find an equivalent ARIN page of "how much we've allocated from >> our last /8" - the statistics show that just over 2x /16s worth have >> been assigned/allocated between January 2016 and July 2016, so a lower >> rate by some margin than RIPE - but there are of course policy >> differences at play there. > > > ARIN's last /8 was run to zero last year. I win the d'oh prize for failing to notice that, although I do have some vague recollection of "Hmm, that will be interesting" now that I think about it. This explains why I thought that ARIN allocation graph looked so random. Interesting times. Well, as I said in another post on this thread, lets hope the v6-as-default tipping point comes sooner rather than later. Paul. From cb.list6 at gmail.com Sun Sep 25 17:50:50 2016 From: cb.list6 at gmail.com (Ca By) Date: Sun, 25 Sep 2016 10:50:50 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: On Sunday, September 25, 2016, jim deleskie wrote: > Sorry but you are mistaken. I've worked at Sr. levels for several LARGE > and medium sized networks. > > mazel tov > > What does it cost and what do we make doing it, over rules what is "good > for the internet" every time it came up. > > 100% agree Thats why i want to see a pie chart of attribution. Charter had this, vz had that, and so on. Headline reads "xyz isp totally hacked network overrun with bots takes down journalists...FCC and DHS demand heads role ... congress yells at ceo... investors dump stock" Perhaps release the article to the brass first, with an alternate ate headline "xyz isp seriously commit to security partners to secure critical infrastructure " You have 2 weeks to pick the story > > On Sun, Sep 25, 2016 at 2:27 PM, Ca By > wrote: > >> On Sunday, September 25, 2016, John Kristoff > > wrote: >> >> > On Sun, 25 Sep 2016 14:36:18 +0000 >> > Ca By > > >> wrote: >> > >> > > As long as their is one spoof capable network on the net, the problem >> > will >> > > not be solved. >> > >> > This is not strictly true. If it could be determined where a large >> > bulk of the spoofing came from, public pressure could be applied. This >> > may not have been the issue in this case, but in many amplification and >> > reflection attacks, the originating spoof-enabled networks were from a >> > limited set of networks. De-peering, service termination, shaming, etc >> > could have an effect. >> > >> > John >> > >> >> Ok, sorry for the not being exact. I am trying to be practical. >> >> My point is, a lot of access networks will respond to public pressure if >> the data is exposed on the offending real ips of the iot crap, and they >> will enforce their AUP. >> >> We have seen comcast do just that, on this list a few months back. That >> path has legs. >> >> Google also blocks service to certain hacked networks as well, we have >> seen >> that on this list too. That is an interesting angle in the krebs case. >> Will >> google block service to folks sharing ip with the iot ddos mess ? >> > > From lear at cisco.com Sun Sep 25 18:16:47 2016 From: lear at cisco.com (Eliot Lear) Date: Sun, 25 Sep 2016 20:16:47 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160925120021.79280a95@p50.localdomain> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: <1774394c-d411-7446-6a80-cd247d12d587@cisco.com> Has anyone stopped to consider what a gift these hackers gave all of us? They exposed their capabilities and nobody got hurt. We all had a notion as to what sort of attacks were possible in theory. Now we have reality. Business being what it is, customers may not be interested in others' security, but IoT being what it is, they might be interested in their own: in this instance, as I understand it, cameras were involved. If a camera could be used to attack someone else, it could be used to invade the privacy of the owner. If consumers come to see that as a threat, that'd be a good first step to internalizing what was an externality. At that point you can sell something. Big if, though. Eliot On 9/25/16 7:00 PM, John Kristoff wrote: > On Sun, 25 Sep 2016 14:36:18 +0000 > Ca By wrote: > >> As long as their is one spoof capable network on the net, the problem will >> not be solved. > This is not strictly true. If it could be determined where a large > bulk of the spoofing came from, public pressure could be applied. This > may not have been the issue in this case, but in many amplification and > reflection attacks, the originating spoof-enabled networks were from a > limited set of networks. De-peering, service termination, shaming, etc > could have an effect. > > John > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From rekoil at semihuman.com Sun Sep 25 18:46:40 2016 From: rekoil at semihuman.com (Chris Woodfield) Date: Sun, 25 Sep 2016 11:46:40 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160924144757.6291.qmail@ary.lan> References: <20160924144757.6291.qmail@ary.lan> Message-ID: <73A642B8-1E55-457B-B8CA-61F0EE2531D6@semihuman.com> > On Sep 24, 2016, at 7:47 AM, John Levine wrote: > >>> Well...by anycast, I meant BGP anycast, spreading the "target" >>> geographically to a dozen or more well connected/peered origins. At that >>> point, your ~600G DDoS might only be around >> >> anycast and tcp? the heck you say! :) > > People who've tried it say it works fine. Routes don't flap that often. > There are a number of companies terminating anycasted TCP endpoints without issue. It?s not exactly turnkey, but it?s hardly black magic either. Here?s Nick Holt @Microsoft presenting their experience: https://www.youtube.com/watch?v=40MONHHF2BU -Chris From brandon at rd.bbc.co.uk Sun Sep 25 19:14:30 2016 From: brandon at rd.bbc.co.uk (Brandon Butterworth) Date: Sun, 25 Sep 2016 20:14:30 +0100 (BST) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Message-ID: <201609251914.UAA01211@sunf10.rd.bbc.co.uk> > From: jim deleskie > Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and > medium sized networks. What does it cost and what do we make doing it, > over rules what is "good for the internet" every time it came up. "nice network you have there, shame if something were to happen to it" one day they may be the target themselves then they can explain to shareholders their part in enabling so much business disruption Sadly it seems there will always be an exploding Pinto on the internet Perhaps Akamai could present them with a bill for unwanted traffic as they're monetising ddos they may as well charge both sides and having dropped Krebs due to the disruption a court may agree damages too. brandon From deleskie at gmail.com Sun Sep 25 19:28:30 2016 From: deleskie at gmail.com (jim deleskie) Date: Sun, 25 Sep 2016 15:28:30 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <201609251914.UAA01211@sunf10.rd.bbc.co.uk> Message-ID: <57e82560.67ae6b0a.3f213.a3da@mx.google.com> Brandon, Sorry you don't understand how multinational companies and peering agreements work, nor any of the relationships my past networks would of had with akamai. But be confident in the fact none of your concerns would of been an issue and it certainly wasn't because decisions were made with out all aspects being taken into play -jim ? Original Message ? From:brandon at rd.bbc.co.uk Sent:September 25, 2016 3:16 PM To:cb.list6 at gmail.com; deleskie at gmail.com Cc:nanog at nanog.org; jtk at aharp.iorc.depaul.edu Subject:Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey > From: jim deleskie > Sorry but you are mistaken. I've worked at Sr. levels for several LARGE and > medium sized networks.? What does it cost and what do we make doing it, > over rules what is "good for the internet" every time it came up. "nice network you have there, shame if something were to happen to it" one day they may be the target themselves then they can explain to shareholders their part in enabling so much business disruption Sadly it seems there will always be an exploding Pinto on the internet Perhaps Akamai could present them with a bill for unwanted traffic as they're monetising ddos they may as well charge both sides and having dropped Krebs due to the disruption a court may agree damages too. brandon From brandon at rd.bbc.co.uk Sun Sep 25 20:19:04 2016 From: brandon at rd.bbc.co.uk (Brandon Butterworth) Date: Sun, 25 Sep 2016 21:19:04 +0100 (BST) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Message-ID: <201609252019.VAA04497@sunf10.rd.bbc.co.uk> > From deleskie at gmail.com Sun Sep 25 20:26:56 2016 > Sorry you don't understand how multinational companies and > peering agreements work Right, thanks for letting me know. > nor any of the relationships my past networks would of had with akamai I don't care what yours were in the past, if peering agreements don't account for ddos maybe it's time they changed to do so. ddos protection is getting like anti virus where it's a big enough business the companies selling protection would like it to carry on forever (noting you have a stake in ddos don't take it personally) brandon From la at qrator.net Sun Sep 25 18:48:53 2016 From: la at qrator.net (Alexander Lyamin) Date: Sun, 25 Sep 2016 20:48:53 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160925120021.79280a95@p50.localdomain> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: This time around its not about spoofing. I presume this is development of the same botnet/worm that we seen day2 of Shellshock public disclosure - its was pretty hightech - golang, arm/mips/x86 support, multiple attack vectors - inlcuding (surprisingly) very effective password guessing. It counted ~100k heads on day2, and i suppose they did grew quite a bit. Thats part of a problem why cause that much havoc - they do have real IP addresses and reasonably well conected - so they can wreck a havoc in bandwidth and tcp stack. They most likely do not have enough resources to do Full Browser Stack, thats why I think L7 capabilities of the botnet will be very basic. On Sun, Sep 25, 2016 at 7:00 PM, John Kristoff wrote: > On Sun, 25 Sep 2016 14:36:18 +0000 > Ca By wrote: > > > As long as their is one spoof capable network on the net, the problem > will > > not be solved. > > This is not strictly true. If it could be determined where a large > bulk of the spoofing came from, public pressure could be applied. This > may not have been the issue in this case, but in many amplification and > reflection attacks, the originating spoof-enabled networks were from a > limited set of networks. De-peering, service termination, shaming, etc > could have an effect. > > John > -- Alexander Lyamin CEO | Qrator * Labs* office: 8-800-3333-LAB (522) mob: +7-916-9086122 skype: melanor9 mailto: la at qrator.net From nanog at brettglass.com Sun Sep 25 20:01:49 2016 From: nanog at brettglass.com (Brett Glass) Date: Sun, 25 Sep 2016 14:01:49 -0600 Subject: IP addresses being attacked in Krebs DDoS? Message-ID: <201609252009.OAA17053@mail.lariat.net> As an ISP who is pro-active when it comes to security, I'd like to know what IP address(es) are being hit by the Krebs on Security DDoS attack. If we know, we can warn customers that they are harboring infected PCs and/or IoT devices. (And if all ISPs did this, it would be possible to curtail such attacks and plug the security holes that make them possible.) --Brett Glass, LARIAT.NET From nanog at radu-adrian.feurdean.net Sun Sep 25 20:50:42 2016 From: nanog at radu-adrian.feurdean.net (Radu-Adrian Feurdean) Date: Sun, 25 Sep 2016 22:50:42 +0200 Subject: One Year On: IPv4 Exhaust In-Reply-To: References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> On Sun, Sep 25, 2016, at 18:29, Ca By wrote: > Think it is fair to say big content and big eyeballs have moved to IPv6 > (notable exceptions exist) > > http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ Big, yes, many - not really. While looking in the flow logs I could see the same (bandwidth intensive) destinations again and again. It's like ~4-5 destinations doing at least half of the IPv6 traffic. From nanog at radu-adrian.feurdean.net Sun Sep 25 20:57:39 2016 From: nanog at radu-adrian.feurdean.net (Radu-Adrian Feurdean) Date: Sun, 25 Sep 2016 22:57:39 +0200 Subject: One Year On: IPv4 Exhaust In-Reply-To: <5cc20723-80da-0b21-4f46-86b730223114@rollernet.us> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <5cc20723-80da-0b21-4f46-86b730223114@rollernet.us> Message-ID: <1474837059.4092385.736560425.14D42007@webmail.messagingengine.com> On Sun, Sep 25, 2016, at 19:40, Seth Mattinen wrote: > ARIN's last /8 was run to zero last year. > > Anything since then has been randomness from the waiting list such as: > https://www.arin.net/announcements/2016/20160902.html .... and a slightly more restricted "really last" /10 : 23.128.0.0/10 (so-called "to facilitate IPv6 deployment") .... From johnl at iecc.com Sun Sep 25 21:01:55 2016 From: johnl at iecc.com (John R. Levine) Date: 25 Sep 2016 17:01:55 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> <20160925172609.10758.qmail@ary.lan> Message-ID: > https://www.internetsociety.org/sites/default/files/01_5.pdf > > The attack is triggered by a few spoofs somewhere in the world. It is not > feasible to stop this. That paper is about reflection attacks. From what I've read, this was not a reflection attack. The IoT devices are infected with botware which sends attack traffic directly. Address spoofing is not particularly useful for controlling botnets. For example, the Conficker botnet generated pseudo-random domain names where the bots looked for control traffic. > Please see https://www.ietf.org/rfc/rfc6561.txt Uh, yes, we're familiar with that. We even know the people who wrote it. It could use an update for IoT since I get the impression that in many cases the only way for a nontechnical user to fix the infection is to throw the device away. Regards, John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly From marka at isc.org Sun Sep 25 21:07:25 2016 From: marka at isc.org (Mark Andrews) Date: Mon, 26 Sep 2016 07:07:25 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Sun, 25 Sep 2016 20:48:53 +0200." References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> Message-ID: <20160925210725.A27185504B03@rock.dv.isc.org> This is such a golden opportunity for each of you to find compromised hosts on your network or your customer's network. The number of genuine lookups of the blog vs the number of botted machine would make it almost certain that anything directed at the blog is a compromised machine. A phone call to the customer / further analysis would reduce the false positive rate. Mark In message , Alexander Lyamin writes: > This time around its not about spoofing. > > I presume this is development of the same botnet/worm that we seen day2 of > Shellshock public disclosure - its was pretty hightech - golang, > arm/mips/x86 support, multiple attack vectors - inlcuding (surprisingly) > very effective password guessing. > It counted ~100k heads on day2, and i suppose they did grew quite a bit. > > > Thats part of a problem why cause that much havoc - they do have real IP > addresses and reasonably well conected - so they can wreck a havoc in > bandwidth and tcp stack. > > They most likely do not have enough resources to do Full Browser Stack, > thats why I think L7 capabilities of the botnet will be very basic. > > > > On Sun, Sep 25, 2016 at 7:00 PM, John Kristoff wrote: > > > On Sun, 25 Sep 2016 14:36:18 +0000 > > Ca By wrote: > > > > > As long as their is one spoof capable network on the net, the problem > > will > > > not be solved. > > > > This is not strictly true. If it could be determined where a large > > bulk of the spoofing came from, public pressure could be applied. This > > may not have been the issue in this case, but in many amplification and > > reflection attacks, the originating spoof-enabled networks were from a > > limited set of networks. De-peering, service termination, shaming, etc > > could have an effect. > > > > John > > > > > > -- > > Alexander Lyamin > > CEO | Qrator * Labs* > > office: 8-800-3333-LAB (522) > > mob: +7-916-9086122 > > skype: melanor9 > > mailto: la at qrator.net -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From marka at isc.org Sun Sep 25 21:27:37 2016 From: marka at isc.org (Mark Andrews) Date: Mon, 26 Sep 2016 07:27:37 +1000 Subject: One Year On: IPv4 Exhaust In-Reply-To: Your message of "Sun, 25 Sep 2016 22:50:42 +0200." <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> Message-ID: <20160925212737.3FD045504D95@rock.dv.isc.org> In message <1474836642.4090975.736557521.25674A77 at webmail.messagingengine.com>, "Radu-Adrian Feurdean" writes: > On Sun, Sep 25, 2016, at 18:29, Ca By wrote: > > Think it is fair to say big content and big eyeballs have moved to IPv6 > > (notable exceptions exist) > > > > http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/ > > Big, yes, many - not really. > While looking in the flow logs I could see the same (bandwidth > intensive) destinations again and again. It's like ~4-5 destinations > doing at least half of the IPv6 traffic. But it shows that if you turn on IPv6 on the servers you will get IPv6 traffic. We are no longer is a world where turning on IPv6 got you a handful of connections. There are billions of devices that can talk IPv6 to you today the moment you allow them to. Can all your customers talk IPv6 to you? No. It the proportion of customers that can talk IPv6 to you increasing? Yes. Is somewhere between 11-14% worldwide enough for you to invest the time to turn on IPv6 enough? It should be. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From ryan.landry at gmail.com Sun Sep 25 21:50:01 2016 From: ryan.landry at gmail.com (ryan landry) Date: Sun, 25 Sep 2016 21:50:01 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160925210725.A27185504B03@rock.dv.isc.org> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> Message-ID: On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews wrote: > > This is such a golden opportunity for each of you to find compromised > hosts on your network or your customer's network. The number of > genuine lookups of the blog vs the number of botted machine would > make it almost certain that anything directed at the blog is a > compromised machine. A phone call to the customer / further analysis > would reduce the false positive rate. > > Mark > > i wish you luck with that. explaining to grandma that her samsung smart tv has been rooted and needs to be updated should be good fun. for isp's it's a resourcing vs revenue problem. always has been. always will be. far more inclined to hold liable the folks that are churning out terribly dangerous cpe / IoT(shit). surely some regulatory body is looking into this. From patrick at ianai.net Sun Sep 25 21:50:38 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Sun, 25 Sep 2016 17:50:38 -0400 Subject: IP addresses being attacked in Krebs DDoS? In-Reply-To: <201609252009.OAA17053@mail.lariat.net> References: <201609252009.OAA17053@mail.lariat.net> Message-ID: <581FADF1-B321-44B4-B78F-87C28FD8817B@ianai.net> On Sep 25, 2016, at 4:01 PM, Brett Glass wrote: > As an ISP who is pro-active when it comes to security, I'd like to know what IP address(es) are being hit by the Krebs on Security DDoS attack. If we know, we can warn customers that they are harboring infected PCs and/or IoT devices. (And if all ISPs did this, it would be possible to curtail such attacks and plug the security holes that make them possible.) [Pardon the slightly less than specific details below. Must be careful about disclosing names or information which is not public yet.] What Brett is asking seems reasonable, even useful. Unfortunately, it is not as simple as posting a list of addresses on a website. Many devices are compromised because of default user/pass settings. Publishing a list of IP addresses which are so trivially compromised is handing the miscreants a gift. We have done things like this with open DNS resolvers and open NTP servers. (THANK YOU JARED!!!) However, we had a hope of the administrators fixing the problem, and they were at least somewhat easier to find. This list is different. Harder to find, harder to fix. Grandma is unlikely to think about logging into her webcam and changing the admin password - to say nothing of reading NANOG in the first place. Hell, even if she did, how exactly do you remove malware from a SmartTV? Obviously we do not consider Brett a bad actor. It is likely we can work something out with ISPs like Brett and give them the addresses on their network which need remediation. But this is not a five minute job. Plus most of the people working on this do so in their spare time. So please be patient as the lists are gathered, sorted, and offered in a reasonable manner. If you are a member of the various secops lists, more info will be forthcoming. If not, I?m sure someone will make information available in wider channels. To be clear, I am not doing this work personally, so do not email me. The people who are doing this work deserve a hearty and huge thanks from the community. If you know one of them, buy them a drink or dinner, or at least give them a hug. :) I know I will be doing so in Dallas if they let me. -- TTFN, patrick From patrick at ianai.net Sun Sep 25 21:57:42 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Sun, 25 Sep 2016 17:57:42 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> Message-ID: <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> On Sep 25, 2016, at 5:50 PM, ryan landry wrote: > On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews wrote: >> This is such a golden opportunity for each of you to find compromised >> hosts on your network or your customer's network. The number of >> genuine lookups of the blog vs the number of botted machine would >> make it almost certain that anything directed at the blog is a >> compromised machine. A phone call to the customer / further analysis >> would reduce the false positive rate. >> >> Mark >> >> > i wish you luck with that. explaining to grandma that her samsung smart tv > has been rooted and needs to be updated should be good fun. > > for isp's it's a resourcing vs revenue problem. always has been. always > will be. far more inclined to hold liable the folks that are churning out > terribly dangerous cpe / IoT(shit). surely some regulatory body is looking > into this. Yeah, ?cause that was so successful in the past. Remember University of Wisconsin vs. D-Link and their hard-coded NTP server address? -- TTFN, patrick From nanog at radu-adrian.feurdean.net Sun Sep 25 21:58:10 2016 From: nanog at radu-adrian.feurdean.net (Radu-Adrian Feurdean) Date: Sun, 25 Sep 2016 23:58:10 +0200 Subject: One Year On: IPv4 Exhaust In-Reply-To: <20160925212737.3FD045504D95@rock.dv.isc.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> <20160925212737.3FD045504D95@rock.dv.isc.org> Message-ID: <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> On Sun, Sep 25, 2016, at 23:27, Mark Andrews wrote: > But it shows that if you turn on IPv6 on the servers you will get > IPv6 traffic. We are no longer is a world where turning on IPv6 > got you a handful of connections. There are billions of devices > that can talk IPv6 to you today the moment you allow them to. I know, but for the "server guys" turning on IPv6 it's pretty low on priority list. > Can all your customers talk IPv6 to you? No. > It the proportion of customers that can talk IPv6 to you increasing? > Yes. My customers are eyeballs. Residential ones have dual-stack by default, business - some have, some don't and some explicitly refuse (or ask for v6 to be disabled). > Is somewhere between 11-14% worldwide enough for you to invest the > time to turn on IPv6 enough? It should be. Since they (the 11-14% worldwide) do have IPv4 anyway, some consider it's not worth; at least not yet. The issue with IPv6 deployment it's not as simple as some people suggest. It's not a technical problem either, but it's a big one. From baldur.norddahl at gmail.com Sun Sep 25 21:59:20 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Sun, 25 Sep 2016 23:59:20 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> Message-ID: > i wish you luck with that. explaining to grandma that her samsung smart tv > has been rooted and needs to be updated should be good fun. The sad thing is that if we boot out grandma they will just switch to one of our competors and the TV will still be a bot. You can't win. From nick at foobar.org Sun Sep 25 22:26:48 2016 From: nick at foobar.org (Nick Hilliard) Date: Sun, 25 Sep 2016 23:26:48 +0100 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> Message-ID: <57E84F28.2060207@foobar.org> Baldur Norddahl wrote: > The sad thing is that if we boot out grandma they will just switch to one > of our competors and the TV will still be a bot. You can't win. Good thing the smart TV / other IoT manufacturers have taken the responsible approach and have committed to providing lifetime software updates for all the Internet-connected devices they manufacture. Nick From owen at delong.com Sun Sep 25 22:28:08 2016 From: owen at delong.com (Owen DeLong) Date: Sun, 25 Sep 2016 16:28:08 -0600 Subject: One Year On: IPv4 Exhaust In-Reply-To: <57E7F8F5.6070402@prt.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> Message-ID: <07B9C616-C017-43AB-9BC5-3A8FFDFC6A64@delong.com> > On Sep 25, 2016, at 10:19 AM, Paul Thornton wrote: > > On 25/09/2016 01:54, Jay R. Ashworth wrote: >> One year ago today, at 12:36pm EDT, Facebook On This Day reminds me, John >> Curran announced that the last IPv4 address block in ARIN's Free Pool had >> been assigned. >> >> How's that been workin' out for everyone? > > If you'll all indulge a bit of a RIPE-centric reply on this; I've was allocated a /22 from around half-way through 185.169.0.0/16 last week (185 being RIPE's final /8). > > Assuming that RIPE are allocating sequentially - and I believe they are - This means that they have consumed around 66.5% of their final /8. They started allocating from this in September 2012, which suggests a reasonably low consumption rate but the RIPE final /8 will be exhausted in around two years time. > > I can't find an equivalent ARIN page of "how much we've allocated from our last /8" - the statistics show that just over 2x /16s worth have been assigned/allocated between January 2016 and July 2016, so a lower rate by some margin than RIPE - but there are of course policy differences at play there. The reason you can?t find such a thing is because ARIN doesn?t have a last /8 policy, per se, like RIPE and APNIC. Instead, ARIN set aside blocks well before the last /8 for critical infrastructure (Key high-level name servers, IXPs, etc.) and IPv6 transition. The IPv6 transition space has a pretty limited set of valid use cases as does the critical infrastructure block, so ARIN is probably allocating those relatively slowly, but they aren?t coming from the ?last /8?, to the best of my knowledge. The last /8 was allocated business as usual from the free pool and may well have provided the last allocation from the ?virgin free pool? (as opposed to reclaimed blocks). > > Now the operational question of "How has this affected us" is probably best answered with "We've had to pay real money for IPv4 addresses since then." What may be much more interesting is what happens when the fairly ready supply of IPv4 addresses in the secondary transfer market starts to dry up. Just throwing additional money at the problem will probably not be an effective or viable solution then. IMHO, sane organizations see this writing on the walls and are deploying IPv6 at an increasing rate. If people act at a responsible pace, they should be able to get IPv6 deployed before we run out of readily available secondary market supply. If not, then, well, it?s not like they didn?t have 20+ years warning so I don?t exactly feel a great deal of sympathy for their self-inflicted wound(s). > > I'm sure that Geoff Huston has a much more accurate and colourful set of predictions than my back-of-envelope calculations for those interested! Yep. IPv6 is the present. IPv4 is the past. The sooner we get more networks to regard the world in this way, the quicker life gets better for everyone. Owen From nanog at brettglass.com Sun Sep 25 22:35:18 2016 From: nanog at brettglass.com (Brett Glass) Date: Sun, 25 Sep 2016 16:35:18 -0600 Subject: IP addresses being attacked in Krebs DDoS? In-Reply-To: <581FADF1-B321-44B4-B78F-87C28FD8817B@ianai.net> References: <201609252009.OAA17053@mail.lariat.net> <581FADF1-B321-44B4-B78F-87C28FD8817B@ianai.net> Message-ID: <201609252235.QAA17953@mail.lariat.net> At 03:50 PM 9/25/2016, Patrick W. Gilmore wrote: >What Brett is asking seems reasonable, even useful. Unfortunately, >it is not as simple as posting a list of addresses on a website. > >Many devices are compromised because of default user/pass >settings. Publishing a list of IP addresses which are so trivially >compromised is handing the miscreants a gift. I think you may have misunderstood my request. I am not asking for the IP addresses of the bots, but the address or addresses which they are attacking. I can then scan outgoing packets for those destination addresses, and -- if I see them -- work my way back to the customers who are unknowingly harboring infected devices. Those devices could be PCs, Webcams, DVRs, even thermostats.... The customers may not know that they have changeable passwords or backdoors. By doing this, we can not only enhance our users' security but forestall complaints. We have had more than one customer quit because an infected device on his or her network impacted the quality of video streaming or VoIP... and, of course, he blamed the ISP. Everyone ALWAYS blames the ISP. ;-) --Brett Glass From damian at google.com Sun Sep 25 22:40:58 2016 From: damian at google.com (Damian Menscher) Date: Sun, 25 Sep 2016 15:40:58 -0700 Subject: IP addresses being attacked in Krebs DDoS? In-Reply-To: <201609252009.OAA17053@mail.lariat.net> References: <201609252009.OAA17053@mail.lariat.net> Message-ID: On Sun, Sep 25, 2016 at 1:01 PM, Brett Glass wrote: > As an ISP who is pro-active when it comes to security, I'd like to know > what IP address(es) are being hit by the Krebs on Security DDoS attack. If > we know, we can warn customers that they are harboring infected PCs and/or > IoT devices. (And if all ISPs did this, it would be possible to curtail > such attacks and plug the security holes that make them possible.) > 130.211.45.45 (it's just the one IP, not DNS-balanced). Thanks for your interest in cleaning up your infected customers! 10,000 ASNs to go.... Damian -- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169 From owen at delong.com Sun Sep 25 22:41:17 2016 From: owen at delong.com (Owen DeLong) Date: Sun, 25 Sep 2016 16:41:17 -0600 Subject: One Year On: IPv4 Exhaust In-Reply-To: <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> <20160925212737.3FD045504D95@rock.dv.isc.org> <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> Message-ID: <50948D94-134D-4717-80CC-08DF27E90802@delong.com> > On Sep 25, 2016, at 3:58 PM, Radu-Adrian Feurdean wrote: > > On Sun, Sep 25, 2016, at 23:27, Mark Andrews wrote: > >> But it shows that if you turn on IPv6 on the servers you will get >> IPv6 traffic. We are no longer is a world where turning on IPv6 >> got you a handful of connections. There are billions of devices >> that can talk IPv6 to you today the moment you allow them to. > > I know, but for the "server guys" turning on IPv6 it's pretty low on > priority list. Which is a selfish, arrogant, and extremely short-sighted and unenlightened view of self-interest. (see below) > >> Can all your customers talk IPv6 to you? No. >> It the proportion of customers that can talk IPv6 to you increasing? >> Yes. > > My customers are eyeballs. Residential ones have dual-stack by default, > business - some have, some don't and some explicitly refuse (or ask for > v6 to be disabled). If you don?t want to face an escalating nightmare for supporting those businesses in the last category in the future, you should probably be educating them today. Sure, go ahead and do what they want, but at least make a stab at letting them know why this might not be such a great idea going forward. > >> Is somewhere between 11-14% worldwide enough for you to invest the >> time to turn on IPv6 enough? It should be. > > Since they (the 11-14% worldwide) do have IPv4 anyway, some consider > it's not worth; at least not yet. This is a circular argument? The 11-14% still have IPv4 through various increasingly fragile and unscalable mechanisms mainly to deal with servers that haven?t deployed IPv6 yet. If all the servers they want to reach had IPv6, it would be relatively easy and highly desirable for their ISPs to turn off their IPv4 relatively quickly. OTOH, the server guys (mostly) can?t get to pure IPv6 because of the lagging eyeball networks that don?t universally deploy IPv6 to all of their customers. It?s like a perverse form of constructive resonance where each one feeds on the other in an escalating destructive cycle. Unfortunately, the ones suffering are not the ones causing the problem, so it becomes another typical example of what is classically known as the ?toxic polluter? problem of capitalist economies. (Absent regulation or morality, dump your toxic waste in such a location as it doesn?t cause you a problem, without regard to the impact on others is the most cost effective solution to the problem) > The issue with IPv6 deployment it's not as simple as some people > suggest. It's not a technical problem either, but it's a big one. For the vast majority of networks, it?s not a big problem, but it hasn?t achieved adequate visibility as a business continuity risk, so it continues to plod along and laggards continue to inflict remote damage. The good news is that as more and more of the larger content and eyeball networks deploy more and more IPv6, the remaining laggards will rapidly become less and less relevant until it?s no longer worth holding up progress on the internet just for the sake of keeping them connected. They will become a series of disconnected IPv4 islands in an IPv6 ocean that passes them by as they sail off into obscurity. Owen From patrick at ianai.net Sun Sep 25 22:57:29 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Sun, 25 Sep 2016 18:57:29 -0400 Subject: IP addresses being attacked in Krebs DDoS? In-Reply-To: <201609252235.QAA17953@mail.lariat.net> References: <201609252009.OAA17053@mail.lariat.net> <581FADF1-B321-44B4-B78F-87C28FD8817B@ianai.net> <201609252235.QAA17953@mail.lariat.net> Message-ID: <1FCF074E-9864-471E-857F-DD521E51EA23@ianai.net> On Sep 25, 2016, at 6:35 PM, Brett Glass wrote: > At 03:50 PM 9/25/2016, Patrick W. Gilmore wrote: >> What Brett is asking seems reasonable, even useful. Unfortunately, it is not as simple as posting a list of addresses on a website. >> >> Many devices are compromised because of default user/pass settings. Publishing a list of IP addresses which are so trivially compromised is handing the miscreants a gift. > > I think you may have misunderstood my request. I am not asking for the IP addresses of the bots, but the address or addresses which they are attacking. I can then scan outgoing packets for those destination addresses, and -- if I see them -- work my way back to the customers who are unknowingly harboring infected devices. Those devices could be PCs, Webcams, DVRs, even thermostats.... The customers may not know that they have changeable passwords or backdoors. > > By doing this, we can not only enhance our users' security but forestall complaints. We have had more than one customer quit because an infected device on his or her network impacted the quality of video streaming or VoIP... and, of course, he blamed the ISP. Everyone ALWAYS blames the ISP. ;-) I did read it the other way. It?s his website, which you can read about on ? his website, http://krebsonsecurity.com/. (And for everyone on this list, it should be trivial to figure out who helped him get the website back up.) Or his twitter feed. Or lots of articles about it. Or lots of mailing lists. Or ? etc. -- TTFN, patrick From list at satchell.net Sun Sep 25 22:59:15 2016 From: list at satchell.net (Stephen Satchell) Date: Sun, 25 Sep 2016 15:59:15 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> Message-ID: <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> On 09/25/2016 07:32 AM, Jay R. Ashworth wrote: >> From: "Jay Farrell via NANOG" >> > And of course Brian Krebs has a thing or two to say, not the least is which >> > to push for BCP38 (good luck with that, right?). >> > >> > https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ > Well, given how few contributions we've gotten at bcp38.info in the last, > what, 4 years, yeah, I guess so... > Yeah, right. I looked at BCP38.info, and there is very little concrete information. I've been slogging through the two RFCs, 2827 and 3794, and find it tough sledding to extract the nuggets to put into my firewall and routing table. One of the more interesting new additions to my systems is this, to the routing tables: ip route add blackhole 0.0.0.0/32 # broadcast (deprecated) ip route add blackhole 0.0.0.0/8 # ?this? ip route add blackhole 10.0.0.0/8 # private network ip route add blackhole 100.64.0.0/10 # carrier-grade NAT ip route add blackhole 127.0.0.0/8 # loopback ip route add blackhole 169.254.0.0/16 # link-local ip route add blackhole 172.16.0.0/12 # private network ip route add blackhole 192.0.0.0/24 # IANA special purpose registry ip route add blackhole 192.0.2.0/24 # TEST-NET ip route add blackhole 192.88.99.0/24 # 6to4 anycast relay (optional) ip route add blackhole 192.168.0.0/16 # private network ip route add blackhole 198.18.0.0/15 # inter-network testing ip route add blackhole 198.51.100.0/24 # TEST-NET-2 ip route add blackhole 203.0.113.0/24 # TEST-NET-3 ip route add blackhole 224.0.0.0/4 # Multicast (all the implied routes from interface definitions that are inserted automatically into the routing table by the operating system) ip route add via dev ip route add via dev (Has this been published anywhere before? I haven't found any yet.) Notice I've omitted 255.255.255.255, because I have quite a bit of software that uses this broadcast address, and it needs to get onto my intranets; I just have to be sure that the edge link doesn't let it out to the world. One such program that uses broadcast is Dropbox. This routing table addition forms the nucleus from which I can derive most of the source-address input packet filtering (adding 255.255.255.255), as well as destination-address output filtering with code, instead of having to compile the rules by hand. Then there are other packet filters: * TCP ?Lamp test/XMAS? et. al. (mask, set) --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE --tcp-flags FIN,SYN,RST,ACK,URG FIN,SYN,RST,ACK,URG --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG --tcp-flags FIN,ACK FIN --tcp-flags PSH,ACK PSH --tcp-flags URG,ACK URG --tcp-flags SYN,FIN SYN,FIN --tcp-flags SYN,RST SYN,RST --tcp-flags FIN,RST FIN,RST * DNS flood * DNS amplification * NTP flood * NTP amplification * ICMP flood * IGRP flood * protocol/port ?small services? connections And I'm just getting started. It helps that I have a few books talking about the design of firewalls that discuss some of these filters. For DNS amplification, I'm using these IPTABLES rules: > # Limit UDP DNS "root NS" queries (must come before generic stream ACCEPT) > # 012000010000000000010000020001 > -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|012000010000000000010000020001|" --algo bm --from 20 --to 1550 -m recent --set --name dnsrootquery --rsource > -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|012000010000000000010000020001|" --algo bm --from 20 --to 1550 -m recent --rcheck --seconds 20 --hitcount 3 --name dnsrootquery --rsource -m comment --comment "UDP DNS ROOT flood" -j DROP (Another possibility would be to read the DNS hints file for all the root servers, and impose rate-limiting to those IP addresses...but I didn't think of that before and so it's not part of the current DNS server firewall. But the filters above seem to work quite well -- it's been a long time since my ISP upstream's uplink has been nailed up solid.) In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY filtering system -- BSD, Linux, Cisco. From marka at isc.org Sun Sep 25 23:01:49 2016 From: marka at isc.org (Mark Andrews) Date: Mon, 26 Sep 2016 09:01:49 +1000 Subject: One Year On: IPv4 Exhaust In-Reply-To: Your message of "Sun, 25 Sep 2016 23:58:10 +0200." <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> <20160925212737.3FD045504D95@rock.dv.isc.org> <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> Message-ID: <20160925230150.1A863550560D@rock.dv.isc.org> In message <1474840690.4107784.736591409.28E807DF at webmail.messagingengine.com>, "Radu-Adrian Feurdean" writes: > On Sun, Sep 25, 2016, at 23:27, Mark Andrews wrote: > > > But it shows that if you turn on IPv6 on the servers you will get > > IPv6 traffic. We are no longer is a world where turning on IPv6 > > got you a handful of connections. There are billions of devices > > that can talk IPv6 to you today the moment you allow them to. > > I know, but for the "server guys" turning on IPv6 it's pretty low on > priority list. Are those server guys interested in stopping attacks without collateral damage? You can't say that a IPv4 address == 1 customer today. Any protection measures you put in place based on IPv4 addresses are likely to affect more than one customer. > > Can all your customers talk IPv6 to you? No. > > It the proportion of customers that can talk IPv6 to you increasing? > > Yes. > > My customers are eyeballs. Residential ones have dual-stack by default, > business - some have, some don't and some explicitly refuse (or ask for > v6 to be disabled). Lots of residentual customers don't have a unshared IPv4 address. The only reason you are seeing IPv4 from them is that the ISP has had to spend money working around the sheer lazyness of content providers in not providing IPv6. > > Is somewhere between 11-14% worldwide enough for you to invest the > > time to turn on IPv6 enough? It should be. > > Since they (the 11-14% worldwide) do have IPv4 anyway, some consider > it's not worth; at least not yet. Actually almost all of the world does not have complete IPv4, they have a subset of IPv4. You have just got used to not having complete IPv4. > The issue with IPv6 deployment it's not as simple as some people > suggest. It's not a technical problem either, but it's a big one. In most cases it is just a matter of turning it on. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From owen at delong.com Mon Sep 26 00:19:22 2016 From: owen at delong.com (Owen DeLong) Date: Sun, 25 Sep 2016 18:19:22 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160924144757.6291.qmail@ary.lan> References: <20160924144757.6291.qmail@ary.lan> Message-ID: > On Sep 24, 2016, at 8:47 AM, John Levine wrote: > >>> Well...by anycast, I meant BGP anycast, spreading the "target" >>> geographically to a dozen or more well connected/peered origins. At that >>> point, your ~600G DDoS might only be around >> >> anycast and tcp? the heck you say! :) > > People who've tried it say it works fine. Routes don't flap that often. It?s not just about route flap. Imagine the following. For any two any cast points A,B, one can draw a simple Venn diagram of two circles with equal radii overlapping to form an OGIVE. Consider that everyone in the nonintersecting portion of circle A will reach server A without issue. Likewise, everyone in the nonintersecting portion of circle B will reach server B without issue. However, for some subset of those within the OGIVE, it?s entirely likely that they will, instead, be broken by ECMP to both A and B. Here?s where it gets tricky? The people running A and B are unlikely to ever know because of the layers between the end user trapped in the OGIVE and the people running A and B. Most likely, the end users will suffer in silence or go to another website for their needs. If this is a small enough fraction of users, then it won?t be statistically noticeable drop in overall traffic and A,B may never know. For those few end-users that may actually attempt to resolve the issue in some meaningful way, most likely they will call their ISP rather than the administrators of A,B and if their ISP does anything, rather than bug A,B, they will most likely simple make routing more deterministic for this site for this end-user. This is the nature of any cast and how any cast problems with TCP get solved (or don?t in most cases). It?s safe to ignore the silent minority that cannot really tell what is happening in most cases, but that doesn?t mean it ?works? for any standard I would consider valid. Owen From marka at isc.org Mon Sep 26 00:32:17 2016 From: marka at isc.org (Mark Andrews) Date: Mon, 26 Sep 2016 10:32:17 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Sun, 25 Sep 2016 18:19:22 -0600." References: <20160924144757.6291.qmail@ary.lan> Message-ID: <20160926003218.2F7075505E5D@rock.dv.isc.org> In message , Owen DeLong writes: > > > On Sep 24, 2016, at 8:47 AM, John Levine wrote: > > > >>> Well...by anycast, I meant BGP anycast, spreading the "target" > >>> geographically to a dozen or more well connected/peered origins. At > that > >>> point, your ~600G DDoS might only be around > >> > >> anycast and tcp? the heck you say! :) > > > > People who've tried it say it works fine. Routes don't flap that often. > > It???s not just about route flap. > > Imagine the following. For any two any cast points A,B, one can draw a > simple Venn diagram of two circles with equal radii overlapping to form > an OGIVE. > > Consider that everyone in the nonintersecting portion of circle A will > reach server A without issue. > Likewise, everyone in the nonintersecting portion of circle B will reach > server B without issue. > However, for some subset of those within the OGIVE, it???s entirely likely > that they will, instead, be broken by ECMP to both A and B. > > Here???s where it gets tricky??? > > The people running A and B are unlikely to ever know because of the > layers between the end user trapped in the OGIVE and the people running A > and B. Most likely, the end users will suffer in silence or go to another > website for their needs. If this is a small enough fraction of users, > then it won???t be statistically noticeable drop in overall traffic and A,B > may never know. For those few end-users that may actually attempt to > resolve the issue in some meaningful way, most likely they will call > their ISP rather than the administrators of A,B and if their ISP does > anything, rather than bug A,B, they will most likely simple make routing > more deterministic for this site for this end-user. > > This is the nature of any cast and how any cast problems with TCP get > solved (or don???t in most cases). > > It???s safe to ignore the silent minority that cannot really tell what is > happening in most cases, but that doesn???t mean it ???works??? for any > standard I would consider valid. > > Owen Which is why sane operators carefully choose where they deploy ECMP or include hashes of the source and destination addresses into the distribution of traffic over otherwise equal paths. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From owen at delong.com Mon Sep 26 00:45:57 2016 From: owen at delong.com (Owen DeLong) Date: Sun, 25 Sep 2016 18:45:57 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160926003218.2F7075505E5D@rock.dv.isc.org> References: <20160924144757.6291.qmail@ary.lan> <20160926003218.2F7075505E5D@rock.dv.isc.org> Message-ID: <91EAAA58-6E9A-418D-8157-594A815BB5E4@delong.com> Assuming all transit providers your packets may traverse on the way to all of your customers is the kind of thing that leads to me quoting Mr. Bush? ?I encourage my competitors to try this.? Owen > On Sep 25, 2016, at 6:32 PM, Mark Andrews wrote: > > > In message , Owen DeLong writes: >> >>> On Sep 24, 2016, at 8:47 AM, John Levine wrote: >>> >>>>> Well...by anycast, I meant BGP anycast, spreading the "target" >>>>> geographically to a dozen or more well connected/peered origins. At >> that >>>>> point, your ~600G DDoS might only be around >>>> >>>> anycast and tcp? the heck you say! :) >>> >>> People who've tried it say it works fine. Routes don't flap that often. >> >> It?s not just about route flap. >> >> Imagine the following. For any two any cast points A,B, one can draw a >> simple Venn diagram of two circles with equal radii overlapping to form >> an OGIVE. >> >> Consider that everyone in the nonintersecting portion of circle A will >> reach server A without issue. >> Likewise, everyone in the nonintersecting portion of circle B will reach >> server B without issue. >> However, for some subset of those within the OGIVE, it?s entirely likely >> that they will, instead, be broken by ECMP to both A and B. >> >> Here?s where it gets tricky? >> >> The people running A and B are unlikely to ever know because of the >> layers between the end user trapped in the OGIVE and the people running A >> and B. Most likely, the end users will suffer in silence or go to another >> website for their needs. If this is a small enough fraction of users, >> then it won?t be statistically noticeable drop in overall traffic and A,B >> may never know. For those few end-users that may actually attempt to >> resolve the issue in some meaningful way, most likely they will call >> their ISP rather than the administrators of A,B and if their ISP does >> anything, rather than bug A,B, they will most likely simple make routing >> more deterministic for this site for this end-user. >> >> This is the nature of any cast and how any cast problems with TCP get >> solved (or don?t in most cases). >> >> It?s safe to ignore the silent minority that cannot really tell what is >> happening in most cases, but that doesn?t mean it ?works? for any >> standard I would consider valid. >> >> Owen > > Which is why sane operators carefully choose where they deploy ECMP > or include hashes of the source and destination addresses into the > distribution of traffic over otherwise equal paths. > > Mark > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From johnl at iecc.com Mon Sep 26 01:32:40 2016 From: johnl at iecc.com (John R. Levine) Date: 25 Sep 2016 21:32:40 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> Message-ID: > It?s safe to ignore the silent minority that cannot really tell what is happening in most cases, but that doesn?t mean it ?works? for any standard I would consider valid. Huh. So you're saying Bill Woodcock doesn't have the skills to see how his traffic is failing? Regards, John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly From hugo at slabnet.com Mon Sep 26 03:54:00 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Sun, 25 Sep 2016 20:54:00 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <1835846730.157.1474815704457.JavaMail.zimbra@baylink.com> <20160925172609.10758.qmail@ary.lan> Message-ID: <20160926035400.e5t5knfgdgenpi7i@slab-wks-04.int.slabnet.com> On Sun 2016-Sep-25 17:01:55 -0400, John R. Levine wrote: >>https://www.internetsociety.org/sites/default/files/01_5.pdf >> >>The attack is triggered by a few spoofs somewhere in the world. It is not >>feasible to stop this. > >That paper is about reflection attacks. From what I've read, this was >not a reflection attack. The IoT devices are infected with botware >which sends attack traffic directly. Address spoofing is not particularly >useful for controlling botnets. But that's not only remaining use of source address spoofing in direct attacks, no? Even if reflection and amplification are not used, spoofing can still be used for obfuscation. >For example, the Conficker botnet generated pseudo-random domain names >where the bots looked for control traffic. > >>Please see https://www.ietf.org/rfc/rfc6561.txt > >Uh, yes, we're familiar with that. We even know the people who wrote >it. It could use an update for IoT since I get the impression that in >many cases the only way for a nontechnical user to fix the infection >is to throw the device away. > >Regards, >John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", >Please consider the environment before reading this e-mail. https://jl.ly -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From hugo at slabnet.com Mon Sep 26 04:19:31 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Sun, 25 Sep 2016 21:19:31 -0700 Subject: BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ] In-Reply-To: <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> Message-ID: <20160926041931.56loourte4eu6me5@slab-wks-04.int.slabnet.com> On Sun 2016-Sep-25 15:59:15 -0700, Stephen Satchell wrote: >On 09/25/2016 07:32 AM, Jay R. Ashworth wrote: >>>From: "Jay Farrell via NANOG" >>>> And of course Brian Krebs has a thing or two to say, not the least is which >>>> to push for BCP38 (good luck with that, right?). >>>> >>>> https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ >>Well, given how few contributions we've gotten at bcp38.info in the last, >>what, 4 years, yeah, I guess so... >> > >Yeah, right. I looked at BCP38.info, and there is very little >concrete information. I've been slogging through the two RFCs, 2827 >and 3794, and find it tough sledding to extract the nuggets to put >into my firewall and routing table. One of the more interesting new >additions to my systems is this, to the routing tables: > ### snip ### > >In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY >filtering system -- BSD, Linux, Cisco. I am guilty of not yet contributing cookbook-type info to BCP38.info, but: Cisco: http://www.bcp38.info/index.php/HOWTO:Cisco points at http://www.cisco.com/c/en/us/about/security-center/unicast-reverse-path-forwarding.html Juniper: https://www.juniper.net/documentation/en_US/junos14.2/topics/usage-guidelines/interfaces-configuring-unicast-rpf.html http://www.juniper.net/documentation/en_US/junos15.1/topics/topic-map/unicast-rpf.html Linux: From /etc/sysctl.conf: # Uncomment the next two lines to enable Spoof protection (reverse-path # filter) # Turn on Source Address Verification in all interfaces to # prevent some spoofing attacks net.ipv4.conf.default.rp_filter=1 net.ipv4.conf.all.rp_filter=1 Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a thing on Linux. For a belt-and-suspenders approach: If you're running an edge network and not transiting traffic for any other AS, consider using your assigned aggregates prefix lists to filter on egress on your edge for anything not sourced from those aggregates. I'm curious as to the deployment scope and experiences of various sizes of networks in deploying the following: 1. Strict uRPF on customer-facing ports on edge networks 2. Source address filtering on upstream edge egress based on assigned aggregates 3. Destination address filtering on upstream edge ingress based on assigned aggregates -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From lear at cisco.com Mon Sep 26 05:59:48 2016 From: lear at cisco.com (Eliot Lear) Date: Mon, 26 Sep 2016 07:59:48 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> Message-ID: Hi Ryan, On 9/25/16 11:50 PM, ryan landry wrote: > for isp's it's a resourcing vs revenue problem. always has been. Sure. The question is whether IoT can make a change in consumer attitudes. Riek, Bohme, et al have been working on this [1]. And there is earlier work as well. What that earlier work shows, by the way, is that if someone suffers a loss, or even if they know someone who suffers a loss, they'll become considerably more risk averse towards Internet technology, to one extent or another. The Riek analysis doesn't really take into account IoT, by the way. It just looks at losses. But I think the logic is likely to hold as IoT creates more risks. The question is whether the impact will increase, and whether those losses will motivate market opportunities for SPs. I think there's a good chance of that if the solution doesn't involve a vast amount of work on the consumer's part. Eliot [1] "Estimating the costs of consumer-facing cybercrime: A tailored instrument and representative data for six EU countries/", /http://weis2016.econinfosec.org/wp-content/uploads/sites/2/2016/05/WEIS_2016_paper_54-2.pdf -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From Valdis.Kletnieks at vt.edu Mon Sep 26 07:14:44 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Mon, 26 Sep 2016 03:14:44 -0400 Subject: BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ] In-Reply-To: <20160926041931.56loourte4eu6me5@slab-wks-04.int.slabnet.com> References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> <20160926041931.56loourte4eu6me5@slab-wks-04.int.slabnet.com> Message-ID: <91773.1474874084@turing-police.cc.vt.edu> On Sun, 25 Sep 2016 21:19:31 -0700, Hugo Slabbert said: > Linux: > From /etc/sysctl.conf: > > # Uncomment the next two lines to enable Spoof protection (reverse-path=20 > # filter) > # Turn on Source Address Verification in all interfaces to > # prevent some spoofing attacks > net.ipv4.conf.default.rp_filter=1 > net.ipv4.conf.all.rp_filter=1 > > Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a > thing on Linux. See net/ipv6/netfilter/ip6t_rpfilter.c Also, note that a lot of net.ipv4.conf variables also apply to ipv6 (though checking the source tree, this isn't one of them, unless it's via a macro that some quick grepping didn't find...) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From bernat at luffy.cx Mon Sep 26 07:34:34 2016 From: bernat at luffy.cx (Vincent Bernat) Date: Mon, 26 Sep 2016 09:34:34 +0200 Subject: BCP38 deployment [ was Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey ] In-Reply-To: <91773.1474874084@turing-police.cc.vt.edu> (Valdis Kletnieks's message of "Mon, 26 Sep 2016 03:14:44 -0400") References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> <20160926041931.56loourte4eu6me5@slab-wks-04.int.slabnet.com> <91773.1474874084@turing-police.cc.vt.edu> Message-ID: <8760pj867p.fsf@zoro.exoscale.ch> ? 26 septembre 2016 09:14 CEST, Valdis.Kletnieks at vt.edu?: >> Linux: >> From /etc/sysctl.conf: >> >> # Uncomment the next two lines to enable Spoof protection (reverse-path=20 >> # filter) >> # Turn on Source Address Verification in all interfaces to >> # prevent some spoofing attacks >> net.ipv4.conf.default.rp_filter=1 >> net.ipv4.conf.all.rp_filter=1 Only "all" is needed since the kernel will use the max of all and the current interface value. >> Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a >> thing on Linux. > > See net/ipv6/netfilter/ip6t_rpfilter.c > > Also, note that a lot of net.ipv4.conf variables also apply to ipv6 (though > checking the source tree, this isn't one of them, unless it's via a macro that > some quick grepping didn't find...) Yes, it doesn't apply. In Linux, there is no such thing as feature parity for IPv6. davem said in the past that he didn't want this feature in IPv6 and was planning to remove it in IPv4 (but I think this will never happen): http://www.spinics.net/lists/netdev/msg166280.html I am using this instead (assuming ip46tables is iptables + ip6tables): ip46tables -t raw -N RPFILTER ip46tables -t raw -A RPFILTER -m rpfilter -j RETURN iptables -t raw -A RPFILTER -d 255.255.255.255 -p udp --sport bootpc --dport bootps -j RETURN ip6tables -t raw -A RPFILTER -m rpfilter --accept-local -m addrtype --dst-type MULTICAST -j DROP ip46tables -t raw -A RPFILTER -m limit --limit 5/s --limit-burst 5 \ -j NFLOG --nflog-group 99 \ --nflog-prefix "NF: rpfilter: " ip46tables -t raw -A RPFILTER -j DROP ip46tables -t raw -A PREROUTING -j RPFILTER -- Use data arrays to avoid repetitive control sequences. - The Elements of Programming Style (Kernighan & Plauger) From nanog at radu-adrian.feurdean.net Mon Sep 26 10:38:53 2016 From: nanog at radu-adrian.feurdean.net (Radu-Adrian Feurdean) Date: Mon, 26 Sep 2016 12:38:53 +0200 Subject: One Year On: IPv4 Exhaust In-Reply-To: <20160925230150.1A863550560D@rock.dv.isc.org> References: <1319236017.16352.1474764873865.JavaMail.zimbra@baylink.com> <57E7F8F5.6070402@prt.org> <1474836642.4090975.736557521.25674A77@webmail.messagingengine.com> <20160925212737.3FD045504D95@rock.dv.isc.org> <1474840690.4107784.736591409.28E807DF@webmail.messagingengine.com> <20160925230150.1A863550560D@rock.dv.isc.org> Message-ID: <1474886333.1874171.737024025.70223DF1@webmail.messagingengine.com> On Mon, Sep 26, 2016, at 01:01, Mark Andrews wrote: > > In message > <1474840690.4107784.736591409.28E807DF at webmail.messagingengine.com>, > "Radu-Adrian Feurdean" writes: > > > > I know, but for the "server guys" turning on IPv6 it's pretty low on > > priority list. > > Are those server guys interested in stopping attacks without > collateral damage? You can't say that a IPv4 address == 1 customer > today. Any protection measures you put in place based on IPv4 > addresses are likely to affect more than one customer. To put in context, I live and work in France, where NO mobile operator provides IPv6, but they do use CGN. Wired-line operators (some, not all) barely start deploying CGNAT on some of the new customers. Pro/business access operators MUST provide IPv4 in order to be able to survive. Things will probably change, but this is the situation today. So "1 IPv4 = several customers" it's either mobile (with no alternative and separate abuse handling process) or negligible. > > My customers are eyeballs. Residential ones have dual-stack by default, > > business - some have, some don't and some explicitly refuse (or ask for > > v6 to be disabled). > > Lots of residentual customers don't have a unshared IPv4 address. > The only reason you are seeing IPv4 from them is that the ISP has > had to spend money working around the sheer lazyness of content > providers in not providing IPv6. Lots of residential customers still do here. > > > Is somewhere between 11-14% worldwide enough for you to invest the > > > time to turn on IPv6 enough? It should be. > > > > Since they (the 11-14% worldwide) do have IPv4 anyway, some consider > > it's not worth; at least not yet. > > Actually almost all of the world does not have complete IPv4, they > have a subset of IPv4. You have just got used to not having complete > IPv4. > > > The issue with IPv6 deployment it's not as simple as some people > > suggest. It's not a technical problem either, but it's a big one. > > In most cases it is just a matter of turning it on. ... and in some of those cases turning it on is subject to a "change request" that requires validation from some level of management that requests the answers to questions similar to following : "What do we gain from this ? What does it cost to turn on ? What does it cost to support the new feature ?". Giving acceptable answers to people that don't necessarily understand IPv6 (some of them having spent their entire life in "IPv4-only, behind NAT" environments) is not that obvious, and this is the core of the "non-technical problem". You probably don't have to deal a lot with this kind of people.... From maillist at webjogger.net Mon Sep 26 12:52:20 2016 From: maillist at webjogger.net (Adam Greene) Date: Mon, 26 Sep 2016 08:52:20 -0400 Subject: AS4233852001 advertising 192.0.0.0/2? Message-ID: <049201d217f4$d1a6e300$74f4a900$@webjogger.net> We were alerted to this by https://radar.qrator.net. This seems wrong from a number of angles . Adam From job at instituut.net Mon Sep 26 12:59:02 2016 From: job at instituut.net (Job Snijders) Date: Mon, 26 Sep 2016 14:59:02 +0200 Subject: AS4233852001 advertising 192.0.0.0/2? In-Reply-To: <049201d217f4$d1a6e300$74f4a900$@webjogger.net> References: <049201d217f4$d1a6e300$74f4a900$@webjogger.net> Message-ID: <20160926125902.GC3032@Hanna.local> On Mon, Sep 26, 2016 at 08:52:20AM -0400, Adam Greene wrote: > We were alerted to this by https://radar.qrator.net. > > This seems wrong from a number of angles . Maybe the alerting system was confused. I don't see this confirmed in RIPE RIS: https://stat.ripe.net/AS4233852001#tabId=routing There are a number of reasons why the visibility of this announcement could be limited: - AS4233852001 a bogon ASN and as such blocked by filters ala http://as2914.net/bogon_asns/configuration_examples.txt - a /2 is a large prefix, a lot of networks only accept up to /8 Kind regards, Job From nanog-isp at mail.com Mon Sep 26 13:25:38 2016 From: nanog-isp at mail.com (nanog-isp at mail.com) Date: Mon, 26 Sep 2016 15:25:38 +0200 Subject: One Year On: IPv4 Exhaust In-Reply-To: References: <4A85328F-044E-4E47-8C94-3DE177BB2A24@ndsu.edu>, Message-ID: > From: "Owen DeLong" > >> I know, but for the "server guys" turning on IPv6 it's pretty low on >> priority list. > > Which is a selfish, arrogant, and extremely short-sighted and unenlightened view of self-interest. Yes, yes, but is it economically rational? Jared From jtk at depaul.edu Mon Sep 26 13:41:36 2016 From: jtk at depaul.edu (John Kristoff) Date: Mon, 26 Sep 2016 08:41:36 -0500 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> Message-ID: <20160926084136.2fba2404@p50.localdomain> On Sun, 25 Sep 2016 22:59:15 +0000 Stephen Satchell wrote: > In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY > filtering system -- BSD, Linux, Cisco. There is some here for integrating Team Cymru's bogon BGP service into various router platforms: John From shawnl at up.net Mon Sep 26 13:58:58 2016 From: shawnl at up.net (Shawn L) Date: Mon, 26 Sep 2016 09:58:58 -0400 (EDT) Subject: =?utf-8?Q?RE=3A_AS4233852001_advertising_192.0.0.0=2F2=3F?= In-Reply-To: <049201d217f4$d1a6e300$74f4a900$@webjogger.net> References: <049201d217f4$d1a6e300$74f4a900$@webjogger.net> Message-ID: <1474898338.15735778@upnet.mymailsrvr.com> Looks like they're announcing quite a bit -----Original Message----- From: "Adam Greene" Sent: Monday, September 26, 2016 8:52am To: nanog at nanog.org Subject: AS4233852001 advertising 192.0.0.0/2? We were alerted to this by https://radar.qrator.net. This seems wrong from a number of angles . Adam From list at satchell.net Mon Sep 26 14:05:49 2016 From: list at satchell.net (Stephen Satchell) Date: Mon, 26 Sep 2016 07:05:49 -0700 Subject: Request for comment -- BCP38 Message-ID: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> Is this an accurate thumbnail summary of BCP38 (ignoring for the moment the issues of multi-home), or is there something I missed? > The basic philosophy of BCP38 boils down to two axioms: > > Don't let the "bad stuff" into your router > Don't let the "bad stuff" leave your router > > The original definition of "bad stuff" is limited to source- > address grooming both inbound and outbound. I've expanded on the > original definition by including rule generation to control > broadcast address abuse. From fergdawgster at mykolab.com Mon Sep 26 14:11:42 2016 From: fergdawgster at mykolab.com (Paul Ferguson) Date: Mon, 26 Sep 2016 07:11:42 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> Message-ID: <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> No -- BCP38 only prescribes filtering outbound to ensure that no packets leave your network with IP source addresses which are not from within your legitimate allocation. - ferg On September 26, 2016 7:05:49 AM PDT, Stephen Satchell wrote: >Is this an accurate thumbnail summary of BCP38 (ignoring for the moment > >the issues of multi-home), or is there something I missed? > >> The basic philosophy of BCP38 boils down to two axioms: >> >> Don't let the "bad stuff" into your router >> Don't let the "bad stuff" leave your router >> >> The original definition of "bad stuff" is limited to source- >> address grooming both inbound and outbound. I've expanded on the >> original definition by including rule generation to control >> broadcast address abuse. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From math at sizone.org Mon Sep 26 14:47:24 2016 From: math at sizone.org (Ken Chase) Date: Mon, 26 Sep 2016 10:47:24 -0400 Subject: Request for comment -- BCP38 In-Reply-To: <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> Message-ID: <20160926144724.GD15891@sizone.org> This might break some of those badly-behaving "dual ISP" COTS routers out there that use different inbound from outbound paths since each is the fastest of either link. I did this manually when I was messing around with multiple broadband links on a fbsd router years ago, was glad it worked at the time. /kc On Mon, Sep 26, 2016 at 07:11:42AM -0700, Paul Ferguson said: >No -- BCP38 only prescribes filtering outbound to ensure that no packets leave your network with IP source addresses which are not from within your legitimate allocation. > > - ferg > > >On September 26, 2016 7:05:49 AM PDT, Stephen Satchell wrote: >>Is this an accurate thumbnail summary of BCP38 (ignoring for the moment >> >>the issues of multi-home), or is there something I missed? >> >>> The basic philosophy of BCP38 boils down to two axioms: >>> >>> Don't let the "bad stuff" into your router >>> Don't let the "bad stuff" leave your router >>> >>> The original definition of "bad stuff" is limited to source- >>> address grooming both inbound and outbound. I've expanded on the >>> original definition by including rule generation to control >>> broadcast address abuse. > >-- >Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Ken Chase - math at sizone.org Toronto Canada From list at satchell.net Mon Sep 26 14:47:50 2016 From: list at satchell.net (Stephen Satchell) Date: Mon, 26 Sep 2016 07:47:50 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> Message-ID: <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> On 09/26/2016 07:11 AM, Paul Ferguson wrote: > No -- BCP38 only prescribes filtering outbound to ensure that no > packets leave your network with IP source addresses which are not > from within your legitimate allocation. So, to beat that horse to a fare-thee-well, to be BCP38 compliant I need, on every interface sending packets out to the internet, to block any source address matching a subnet in the BOGON list OR not matching any of my routeable network subnets? Plus add null-route entries for all the BOGONs in my routing table so I don't send a bad destination packet to my upstream? From math at sizone.org Mon Sep 26 14:51:51 2016 From: math at sizone.org (Ken Chase) Date: Mon, 26 Sep 2016 10:51:51 -0400 Subject: charges for prefix filter updates (was Re: Any ISPs using AS852 for IP Transit?) In-Reply-To: <20160915182850.GD15995@sizone.org> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <20160915182850.GD15995@sizone.org> Message-ID: <20160926145151.GG26045@sizone.org> Followup: we did the quote/PO/sign-the-order dance. That took about 3-4 days not including our side's lag (which was not insignificant, Im not the guy with the pen). But now it's gone to provisioning and will be a standard *5 days*. Cogent will do this in about 1-6 hours if you provide the LOA's with the request. So will HE. And many others. /kc On Thu, Sep 15, 2016 at 02:28:50PM -0400, Ken Chase said: >I feel this can be a public topic: > >Rogers just charged us that for an update (one update, multiple entries). >We had to go through their quotation machinery too, took like 4-5 days. Additional >time was wasted because we contacted their tech dept directly at the start. (which >is what I do for all my other upstreams...) > >Kinda brutal. > >Cogent and HE nor NAC or Yipes or Tata ever did that to us. > >Nickle and diming -- why, cuz transit is a cheap commodity now, gotta make the >cash somewhere? > >That said Cogent offered us a static /26 along side our BGP years ago then warned >us it'd be $50/mo or something for that # of ips going forward. We didnt need it >so dispensed with it. > >/kc > > >On Thu, Sep 15, 2016 at 02:07:01PM -0400, Jason Lixfeld said: > >If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I???d be interested in hearing from you. > > > >I???d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we???re the only ones??? > > > >Thanks in advance! > >-- >Ken Chase - math at sizone.org Toronto Canada From elmi at 4ever.de Mon Sep 26 14:55:35 2016 From: elmi at 4ever.de (Elmar K. Bins) Date: Mon, 26 Sep 2016 16:55:35 +0200 Subject: Request for comment -- BCP38 In-Reply-To: <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> Message-ID: <20160926145535.GA1505@nbmacvieebi.local> Re Stephen, > So, to beat that horse to a fare-thee-well, to be BCP38 compliant I need, on > every interface sending packets out to the internet, to block any source > address matching a subnet in the BOGON list OR not matching any of my > routeable network subnets? Plus add null-route entries for all the BOGONs > in my routing table so I don't send a bad destination packet to my upstream? The correct way to implement this is - outgoing permit my allocated address blocks as source addresses - outgoing deny EVERYTHING (else) Elmar. From sryan at arbor.net Mon Sep 26 14:57:35 2016 From: sryan at arbor.net (Ryan, Spencer) Date: Mon, 26 Sep 2016 14:57:35 +0000 Subject: charges for prefix filter updates (was Re: Any ISPs using AS852 for IP Transit?) In-Reply-To: <20160926145151.GG26045@sizone.org> References: <97EE8152-6FCE-4650-8C26-0B24C4F67F4D@lixfeld.ca> <20160915182850.GD15995@sizone.org>,<20160926145151.GG26045@sizone.org> Message-ID: I've used HE's tunnelbroker (BGP) a few times to get our ARIN space to a site while waiting on a local carrier to turn up v6, get the proper LOA, etc. I've received better service from the NOC there for a service I didn't pay for than I have from any ISP I've ever given money. They are doing a great job over there. A LOA change (/48-->/40 le 48) took about an hour. Spencer Ryan | Senior Systems Administrator | sryan at arbor.net Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com ________________________________ From: NANOG on behalf of Ken Chase Sent: Monday, September 26, 2016 10:51:51 AM To: Jason Lixfeld Cc: NANOG Subject: Re: charges for prefix filter updates (was Re: Any ISPs using AS852 for IP Transit?) Followup: we did the quote/PO/sign-the-order dance. That took about 3-4 days not including our side's lag (which was not insignificant, Im not the guy with the pen). But now it's gone to provisioning and will be a standard *5 days*. Cogent will do this in about 1-6 hours if you provide the LOA's with the request. So will HE. And many others. /kc On Thu, Sep 15, 2016 at 02:28:50PM -0400, Ken Chase said: >I feel this can be a public topic: > >Rogers just charged us that for an update (one update, multiple entries). >We had to go through their quotation machinery too, took like 4-5 days. Additional >time was wasted because we contacted their tech dept directly at the start. (which >is what I do for all my other upstreams...) > >Kinda brutal. > >Cogent and HE nor NAC or Yipes or Tata ever did that to us. > >Nickle and diming -- why, cuz transit is a cheap commodity now, gotta make the >cash somewhere? > >That said Cogent offered us a static /26 along side our BGP years ago then warned >us it'd be $50/mo or something for that # of ips going forward. We didnt need it >so dispensed with it. > >/kc > > >On Thu, Sep 15, 2016 at 02:07:01PM -0400, Jason Lixfeld said: > >If there are any ISPs who use TELUS/AS852 for IP Transit over BGP, I???d be interested in hearing from you. > > > >I???d like to compare notes to see if you are also paying $250 for each BGP prefix filter updated request, or if we???re the only ones??? > > > >Thanks in advance! > >-- >Ken Chase - math at sizone.org Toronto Canada From fergdawgster at mykolab.com Mon Sep 26 14:58:39 2016 From: fergdawgster at mykolab.com (Paul Ferguson) Date: Mon, 26 Sep 2016 07:58:39 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> Message-ID: <703AEFA2-BCF8-4431-BDB9-681299DC49A0@mykolab.com> > On Sep 26, 2016, at 7:47 AM, Stephen Satchell wrote: > > On 09/26/2016 07:11 AM, Paul Ferguson wrote: >> No -- BCP38 only prescribes filtering outbound to ensure that no >> packets leave your network with IP source addresses which are not >> from within your legitimate allocation. > > So, to beat that horse to a fare-thee-well, to be BCP38 compliant I need, on every interface sending packets out to the internet, to block any source address matching a subnet in the BOGON list OR not matching any of my routeable network subnets? Plus add null-route entries for all the BOGONs in my routing table so I don't send a bad destination packet to my upstream? BCP38 only provides for disallowing spoofed packets into the Internet. Any additional filtering against bosons, etc., are probably a good idea, just not including specifically in BCP38. - ferg ? Paul Ferguson ICEBRG.io Seattle, Washington, USA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 236 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hugo at slabnet.com Mon Sep 26 15:04:05 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 08:04:05 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> Message-ID: <20160926150405.GG16464@bamboo.slabnet.com> On Mon 2016-Sep-26 07:47:50 -0700, Stephen Satchell wrote: >On 09/26/2016 07:11 AM, Paul Ferguson wrote: >>No -- BCP38 only prescribes filtering outbound to ensure that no >>packets leave your network with IP source addresses which are not >>from within your legitimate allocation. > >So, to beat that horse to a fare-thee-well, to be BCP38 compliant I >need, on every interface sending packets out to the internet, to >block any source address matching a subnet in the BOGON list OR not >matching any of my routeable network subnets? TBF, I would assume that you don't have routable/allocated networks within BOGON ranges, so just "if src in mynets permit else discard" covers both sets. >Plus add null-route entries for all the BOGONs in my routing table so I >don't send a bad destination packet to my upstream? I don't think that falls within the purview of BCP38 as BCP38 has to do with source address filtering/verification rather than destination. If you're running full tables and filtering BOGONs on BGP import, though, you shouldn't have routes for BOGONs in your tables and with a 0/0 discard should be dropping them anyway, but if you're not running full tables and so need to "punch holes" in a static default, then explicit null-routes for BOGON destinations would do it. Honestly, though: your upstream probably drops BOGON destinations anyway, so dropping BOGON destinations within your own network is just (a) good hygiene and (b) saves from your transit bill however may bps of BOGON-destined traffic you have. -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From hugo at slabnet.com Mon Sep 26 15:12:54 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 08:12:54 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <20160926144724.GD15891@sizone.org> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> Message-ID: <20160926151254.GH16464@bamboo.slabnet.com> On Mon 2016-Sep-26 10:47:24 -0400, Ken Chase wrote: >This might break some of those badly-behaving "dual ISP" COTS routers out there >that use different inbound from outbound paths since each is the fastest of >either link. As it should. If you have links from both ISP A and ISP B and decide to send traffic out ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* drop that traffic on the floor. There is no automated or scalable way for ISP A to distinguish this "legitimate" use from spoofing; unless you consider it scalable for ISP A to maintain thousands if not more "exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases of customers X, Y, and Z sourcing traffic into ISP A's network using IPs allocated to them by other ISPs? If you want to play asymmetry tricks, get some PI space and make arrangements. If that's outside your wheelhouse, get an ISP that will sell this to you as a service either with dissimilar links they provide to you or over-the-top with tunnels etc. Playing NAT games with different classes of traffic to e.g. send traffic type 1 over ISP A and traffic type 2 over ISP B *BUT* using the corresponding source addresses in each case and having the traffic return back over the same links is fine and dandy. If you send traffic into an ISP-provided link using addresses from another provider, though, that ISP *should* be dropping that traffic. If they don't, send them here so we can yell at them. >I did this manually when I was messing around with multiple broadband links on >a fbsd router years ago, was glad it worked at the time. > >/kc > > >On Mon, Sep 26, 2016 at 07:11:42AM -0700, Paul Ferguson said: > >No -- BCP38 only prescribes filtering outbound to ensure that no packets leave your network with IP source addresses which are not from within your legitimate allocation. > > > > - ferg > > > > > >On September 26, 2016 7:05:49 AM PDT, Stephen Satchell wrote: > >>Is this an accurate thumbnail summary of BCP38 (ignoring for the moment > >> > >>the issues of multi-home), or is there something I missed? > >> > >>> The basic philosophy of BCP38 boils down to two axioms: > >>> > >>> Don't let the "bad stuff" into your router > >>> Don't let the "bad stuff" leave your router > >>> > >>> The original definition of "bad stuff" is limited to source- > >>> address grooming both inbound and outbound. I've expanded on the > >>> original definition by including rule generation to control > >>> broadcast address abuse. > > > >-- > >Sent from my Android device with K-9 Mail. Please excuse my brevity. > >-- >Ken Chase - math at sizone.org Toronto Canada -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From mlm at pixelgate.net Mon Sep 26 15:23:15 2016 From: mlm at pixelgate.net (Mark Milhollan) Date: Mon, 26 Sep 2016 08:23:15 -0700 (PDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> Message-ID: On Sun, 25 Sep 2016, Stephen Satchell wrote: >Yeah, right. I looked at BCP38.info, and there is very little concrete >information. Yeah, it's pretty naked. But how-to isn't the usual stumbling block, as has been pointed out in this thread there needs to be the will to spend resources setting it up and thus committing future resources to maintenance. >I've been slogging through the two RFCs, 2827 and 3794, and find >it tough sledding to extract the nuggets to put into my firewall and routing >table. One of the more interesting new additions to my systems is this, to the >routing tables: A list of martian addresses is useful to avoid sending to or accepting from weird places but it isn't useful for BCP38 purposes, of ensuring a node only uses address(es) assigned to it as the source address in the packets it creates/sends. BGP38 checking is not done by the node itself, though that is not entirely unreasonable. Enforcement is performed by the network as close to the point of the node's attachment as possible -- failures should be discarded or perhaps returned as prohibited and possibly even sampled for use by network staff to work on remediation. In some cases it's very simple and effective to just filter toward your upstream(s), i.e., allow this area's addresses and drop/reject/log the rest. >(Has this been published anywhere before? I haven't found any yet.) Cymru has lists in various formats and levels of (de)aggregation and detail that you can easily turn into those commands, though there's no martians-only lists for IPv6. You might even use one of the "fullbogon" lists to block at a very detailed level if you have sufficient resources or tools that keep needs light, e.g., ipset. >In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY filtering >system -- BSD, Linux, Cisco. For some it's just uRPF, strict or loose as your needs demand, which their router can already perform, e.g., interface your-template/or/range-of-interfaces/etc ip verify unicast reverse-path or ip verify unicast source reachable-via rx or ip verify unicast source reachable-via any Which for Linux is controlled by the net.ipv4.conf.if.rp_filter sysctl key where "if" can be "default", "all" or a specific interface name and a setting of 1 does strict checking while 2 does loose. Or there's always plain old packet filters, with varying degrees of ease or annoyance, as tightly (per customer applied to incoming packets received on their interface) or simply (leaving the pop) as you please and makes sense. /mark From royce at techsolvency.com Mon Sep 26 15:36:30 2016 From: royce at techsolvency.com (Royce Williams) Date: Mon, 26 Sep 2016 07:36:30 -0800 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160924144757.6291.qmail@ary.lan> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <1789387599.132.1474813931145.JavaMail.zimbra@baylink.com> <17c4a69d-127f-6f61-457e-4720bc4c19aa@satchell.net> Message-ID: On Mon, Sep 26, 2016 at 7:23 AM, Mark Milhollan wrote: > > On Sun, 25 Sep 2016, Stephen Satchell wrote: > > >Yeah, right. I looked at BCP38.info, and there is very little concrete > >information. > > Yeah, it's pretty naked. But how-to isn't the usual stumbling block, as > has been pointed out in this thread there needs to be the will to spend > resources setting it up and thus committing future resources to > maintenance. Actually, a clear set of how-tos is the best way to quickly convey -- to busy geeks, and to their busy managers and PMs -- how much that resource spend would probably be. Royce From laszlo at heliacal.net Mon Sep 26 15:47:43 2016 From: laszlo at heliacal.net (Laszlo Hanyecz) Date: Mon, 26 Sep 2016 15:47:43 +0000 Subject: Request for comment -- BCP38 In-Reply-To: <20160926151254.GH16464@bamboo.slabnet.com> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> Message-ID: On 2016-09-26 15:12, Hugo Slabbert wrote: > > On Mon 2016-Sep-26 10:47:24 -0400, Ken Chase wrote: > >> This might break some of those badly-behaving "dual ISP" COTS routers >> out there >> that use different inbound from outbound paths since each is the >> fastest of >> either link. > > As it should. > > If you have links from both ISP A and ISP B and decide to send traffic > out ISP A's link sourced from addresses ISP B allocated to you, ISP A > *should* drop that traffic on the floor. There is no automated or > scalable way for ISP A to distinguish this "legitimate" use from > spoofing; unless you consider it scalable for ISP A to maintain > thousands if not more "exception" ACLs to uRPF and BCP38 egress > filters to cover all of the cases of customers X, Y, and Z sourcing > traffic into ISP A's network using IPs allocated to them by other ISPs? > This is a legitimate and interesting use case that is broken by BCP38. The effectiveness of BCP38 at reducing abuse is dubious, but the benefits of asymmetric routing are well understood. Why should everyone have to go out of their way to break this.. it works fine if you just don't mess with it. > If you want to play asymmetry tricks, get some PI space and make > arrangements. If that's outside your wheelhouse, get an ISP that will > sell this to you as a service either with dissimilar links they > provide to you or over-the-top with tunnels etc. > > Playing NAT games with different classes of traffic to e.g. send > traffic type 1 over ISP A and traffic type 2 over ISP B *BUT* using > the corresponding source addresses in each case and having the traffic > return back over the same links is fine and dandy. If you send > traffic into an ISP-provided link using addresses from another > provider, though, that ISP *should* be dropping that traffic. If they > don't, send them here so we can yell at them. > So instead of being able to use simple destination based routes to direct their traffic, like the service provider can, the CPE operator has to learn and implement policy based routing and manage state to juggle each of the IP addresses they are assigned. It's orders of magnitude harder to do this with the current ecosystem of routers/CPEs, than it is to add a destination route. I think stuff like this is one of the reasons why many are hesitant to implement this type of filtering. It makes a specific type of abuse easier to track down *for someone else* but it doesn't help you much and it can cause debugging nightmares when something doesn't work due to filtering. -Laszlo >> I did this manually when I was messing around with multiple broadband >> links on >> a fbsd router years ago, was glad it worked at the time. >> >> /kc >> >> >> On Mon, Sep 26, 2016 at 07:11:42AM -0700, Paul Ferguson said: >> >No -- BCP38 only prescribes filtering outbound to ensure that no >> packets leave your network with IP source addresses which are not >> from within your legitimate allocation. >> > >> > - ferg >> > >> > >> >On September 26, 2016 7:05:49 AM PDT, Stephen Satchell >> wrote: >> >>Is this an accurate thumbnail summary of BCP38 (ignoring for the >> moment >> >> >> >>the issues of multi-home), or is there something I missed? >> >> >> >>> The basic philosophy of BCP38 boils down to two axioms: >> >>> >> >>> Don't let the "bad stuff" into your router >> >>> Don't let the "bad stuff" leave your router >> >>> >> >>> The original definition of "bad stuff" is limited to source- >> >>> address grooming both inbound and outbound. I've expanded on >> the >> >>> original definition by including rule generation to control >> >>> broadcast address abuse. >> > >> >-- >> >Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> -- >> Ken Chase - math at sizone.org Toronto Canada > From johnl at iecc.com Mon Sep 26 15:56:49 2016 From: johnl at iecc.com (John Levine) Date: 26 Sep 2016 15:56:49 -0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160926035400.e5t5knfgdgenpi7i@slab-wks-04.int.slabnet.com> Message-ID: <20160926155649.14061.qmail@ary.lan> >>That paper is about reflection attacks. From what I've read, this was >>not a reflection attack. The IoT devices are infected with botware >>which sends attack traffic directly. Address spoofing is not particularly >>useful for controlling botnets. > >But that's not only remaining use of source address spoofing in direct >attacks, no? Even if reflection and amplification are not used, spoofing >can still be used for obfuscation. I agree that it would be nice if more networks did ingress filtering, but if you're expecting a major decrease in evil, you will be disappointed. At this point it's mostly useful for identifying the guilty or negligent parties afterwards. R's, John From nanog at ics-il.net Mon Sep 26 16:00:25 2016 From: nanog at ics-il.net (Mike Hammett) Date: Mon, 26 Sep 2016 11:00:25 -0500 (CDT) Subject: Request for comment -- BCP38 In-Reply-To: References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> Message-ID: <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> The only asymmetric routing broken is when the source isn't in public Internet route-able space. That just leaves those multi-ISP WAN routers that NAT it. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Laszlo Hanyecz" To: nanog at nanog.org Sent: Monday, September 26, 2016 10:47:43 AM Subject: Re: Request for comment -- BCP38 On 2016-09-26 15:12, Hugo Slabbert wrote: > > On Mon 2016-Sep-26 10:47:24 -0400, Ken Chase wrote: > >> This might break some of those badly-behaving "dual ISP" COTS routers >> out there >> that use different inbound from outbound paths since each is the >> fastest of >> either link. > > As it should. > > If you have links from both ISP A and ISP B and decide to send traffic > out ISP A's link sourced from addresses ISP B allocated to you, ISP A > *should* drop that traffic on the floor. There is no automated or > scalable way for ISP A to distinguish this "legitimate" use from > spoofing; unless you consider it scalable for ISP A to maintain > thousands if not more "exception" ACLs to uRPF and BCP38 egress > filters to cover all of the cases of customers X, Y, and Z sourcing > traffic into ISP A's network using IPs allocated to them by other ISPs? > This is a legitimate and interesting use case that is broken by BCP38. The effectiveness of BCP38 at reducing abuse is dubious, but the benefits of asymmetric routing are well understood. Why should everyone have to go out of their way to break this.. it works fine if you just don't mess with it. > If you want to play asymmetry tricks, get some PI space and make > arrangements. If that's outside your wheelhouse, get an ISP that will > sell this to you as a service either with dissimilar links they > provide to you or over-the-top with tunnels etc. > > Playing NAT games with different classes of traffic to e.g. send > traffic type 1 over ISP A and traffic type 2 over ISP B *BUT* using > the corresponding source addresses in each case and having the traffic > return back over the same links is fine and dandy. If you send > traffic into an ISP-provided link using addresses from another > provider, though, that ISP *should* be dropping that traffic. If they > don't, send them here so we can yell at them. > So instead of being able to use simple destination based routes to direct their traffic, like the service provider can, the CPE operator has to learn and implement policy based routing and manage state to juggle each of the IP addresses they are assigned. It's orders of magnitude harder to do this with the current ecosystem of routers/CPEs, than it is to add a destination route. I think stuff like this is one of the reasons why many are hesitant to implement this type of filtering. It makes a specific type of abuse easier to track down *for someone else* but it doesn't help you much and it can cause debugging nightmares when something doesn't work due to filtering. -Laszlo >> I did this manually when I was messing around with multiple broadband >> links on >> a fbsd router years ago, was glad it worked at the time. >> >> /kc >> >> >> On Mon, Sep 26, 2016 at 07:11:42AM -0700, Paul Ferguson said: >> >No -- BCP38 only prescribes filtering outbound to ensure that no >> packets leave your network with IP source addresses which are not >> from within your legitimate allocation. >> > >> > - ferg >> > >> > >> >On September 26, 2016 7:05:49 AM PDT, Stephen Satchell >> wrote: >> >>Is this an accurate thumbnail summary of BCP38 (ignoring for the >> moment >> >> >> >>the issues of multi-home), or is there something I missed? >> >> >> >>> The basic philosophy of BCP38 boils down to two axioms: >> >>> >> >>> Don't let the "bad stuff" into your router >> >>> Don't let the "bad stuff" leave your router >> >>> >> >>> The original definition of "bad stuff" is limited to source- >> >>> address grooming both inbound and outbound. I've expanded on >> the >> >>> original definition by including rule generation to control >> >>> broadcast address abuse. >> > >> >-- >> >Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> -- >> Ken Chase - math at sizone.org Toronto Canada > From sethm at rollernet.us Mon Sep 26 16:01:50 2016 From: sethm at rollernet.us (Seth Mattinen) Date: Mon, 26 Sep 2016 09:01:50 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <3da98299-58bd-fee2-168d-56e680a81720@satchell.net> Message-ID: <773e259e-6ec0-0a19-acc4-49d68db6380b@rollernet.us> On 9/26/16 07:47, Stephen Satchell wrote: > On 09/26/2016 07:11 AM, Paul Ferguson wrote: >> No -- BCP38 only prescribes filtering outbound to ensure that no >> packets leave your network with IP source addresses which are not >> from within your legitimate allocation. > > So, to beat that horse to a fare-thee-well, to be BCP38 compliant I > need, on every interface sending packets out to the internet, to block > any source address matching a subnet in the BOGON list OR not matching > any of my routeable network subnets? Plus add null-route entries for > all the BOGONs in my routing table so I don't send a bad destination > packet to my upstream? I start with customer interfaces and configure them to only allow traffic with a source address in their assigned subnet. ~Seth From johnl at iecc.com Mon Sep 26 16:04:33 2016 From: johnl at iecc.com (John Levine) Date: 26 Sep 2016 16:04:33 -0000 Subject: Request for comment -- BCP38 In-Reply-To: <20160926151254.GH16464@bamboo.slabnet.com> Message-ID: <20160926160433.14119.qmail@ary.lan> >If you have links from both ISP A and ISP B and decide to send traffic out >ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >drop that traffic on the floor. There is no automated or scalable way for >ISP A to distinguish this "legitimate" use from spoofing; unless you >consider it scalable for ISP A to maintain thousands if not more >"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >allocated to them by other ISPs? I gather the usual customer response to this is "if you don't want our $50K/mo, I'm sure we can find another ISP who does." >From the conversations I've had with ISPs, the inability to manage legitimate traffic from dual homed customer networks is the most significant bar to widespread BCP38. I realize there's no way to do it automatically now, but it doesn't seem like total rocket science to come up with some way for providers to pass down a signed object to the customer routers that the routers can then pass back up to the customer's other providers. R's, John PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. From nanog at ics-il.net Mon Sep 26 16:15:11 2016 From: nanog at ics-il.net (Mike Hammett) Date: Mon, 26 Sep 2016 11:15:11 -0500 (CDT) Subject: Request for comment -- BCP38 In-Reply-To: <20160926160433.14119.qmail@ary.lan> References: <20160926160433.14119.qmail@ary.lan> Message-ID: <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> Are you talking BGP level customers or individual small businesses' broadband service? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "John Levine" To: nanog at nanog.org Sent: Monday, September 26, 2016 11:04:33 AM Subject: Re: Request for comment -- BCP38 >If you have links from both ISP A and ISP B and decide to send traffic out >ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >drop that traffic on the floor. There is no automated or scalable way for >ISP A to distinguish this "legitimate" use from spoofing; unless you >consider it scalable for ISP A to maintain thousands if not more >"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >allocated to them by other ISPs? I gather the usual customer response to this is "if you don't want our $50K/mo, I'm sure we can find another ISP who does." >From the conversations I've had with ISPs, the inability to manage legitimate traffic from dual homed customer networks is the most significant bar to widespread BCP38. I realize there's no way to do it automatically now, but it doesn't seem like total rocket science to come up with some way for providers to pass down a signed object to the customer routers that the routers can then pass back up to the customer's other providers. R's, John PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. From hugo at slabnet.com Mon Sep 26 16:21:55 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 09:21:55 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> Message-ID: <20160926162155.GI16464@bamboo.slabnet.com> On Mon 2016-Sep-26 11:15:11 -0500, Mike Hammett wrote: >> >>----- Original Message ----- >> >>From: "John Levine" >>To: nanog at nanog.org >>Sent: Monday, September 26, 2016 11:04:33 AM >>Subject: Re: Request for comment -- BCP38 >> >>>If you have links from both ISP A and ISP B and decide to send traffic out >>>ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >>>drop that traffic on the floor. There is no automated or scalable way for >>>ISP A to distinguish this "legitimate" use from spoofing; unless you >>>consider it scalable for ISP A to maintain thousands if not more >>>"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >>>of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >>>allocated to them by other ISPs? >> >>I gather the usual customer response to this is "if you don't want our >>$50K/mo, I'm sure we can find another ISP who does." >> >>From the conversations I've had with ISPs, the inability to manage >>legitimate traffic from dual homed customer networks is the most >>significant bar to widespread BCP38. I realize there's no way to do >>it automatically now, but it doesn't seem like total rocket science to >>come up with some way for providers to pass down a signed object to >>the customer routers that the routers can then pass back up to the >>customer's other providers. >> >>R's, >>John >> >>PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. >> >Are you talking BGP level customers or individual small businesses' >broadband service? I myself am talking about the latter and included the option of PI space to cover that (although I guess at some point this can be made fly with PA space from another provider if both providers are willing enough to play ball), though from the $50/mo figure John listed, I'm assuming he's talking about the latter. Do people really expect to be able to do this on residential or small business broadband networks? I can't remember any time in recent memory where I assumed I could set a source address to any IP I fancy and have that packet successfully make its way through the SP's network. > >----- >Mike Hammett >Intelligent Computing Solutions >http://www.ics-il.com > >Midwest-IX >http://www.midwest-ix.com -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From hugo at slabnet.com Mon Sep 26 16:24:34 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 09:24:34 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <20160926162155.GI16464@bamboo.slabnet.com> References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> <20160926162155.GI16464@bamboo.slabnet.com> Message-ID: <20160926162434.GD3770@bamboo.slabnet.com> On Mon 2016-Sep-26 09:21:55 -0700, Hugo Slabbert wrote: > >On Mon 2016-Sep-26 11:15:11 -0500, Mike Hammett wrote: > >>> >>>----- Original Message ----- >>> >>>From: "John Levine" >>>To: nanog at nanog.org >>>Sent: Monday, September 26, 2016 11:04:33 AM >>>Subject: Re: Request for comment -- BCP38 >>> >>>>If you have links from both ISP A and ISP B and decide to send traffic out >>>>ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >>>>drop that traffic on the floor. There is no automated or scalable way for >>>>ISP A to distinguish this "legitimate" use from spoofing; unless you >>>>consider it scalable for ISP A to maintain thousands if not more >>>>"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >>>>of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >>>>allocated to them by other ISPs? >>> >>>I gather the usual customer response to this is "if you don't want our >>>$50K/mo, I'm sure we can find another ISP who does." >>> >>>From the conversations I've had with ISPs, the inability to manage >>>legitimate traffic from dual homed customer networks is the most >>>significant bar to widespread BCP38. I realize there's no way to do >>>it automatically now, but it doesn't seem like total rocket science to >>>come up with some way for providers to pass down a signed object to >>>the customer routers that the routers can then pass back up to the >>>customer's other providers. >>> >>>R's, >>>John >>> >>>PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. >>> > >>Are you talking BGP level customers or individual small businesses' >>broadband service? > >I myself am talking about the latter... ...dammit...obviously "former" here, not "latter". Caffeine injection requisitioned. >...and included the option of PI space to cover that (although I guess at >some point this can be made fly with PA space from another provider if >both providers are willing enough to play ball), though from the $50/mo >figure John listed, I'm assuming he's talking about the latter. > >Do people really expect to be able to do this on residential or small >business broadband networks? I can't remember any time in recent >memory where I assumed I could set a source address to any IP I fancy >and have that packet successfully make its way through the SP's >network. > >> >>----- >>Mike Hammett >>Intelligent Computing Solutions >>http://www.ics-il.com >> >>Midwest-IX >>http://www.midwest-ix.com -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From aledm at qix.co.uk Mon Sep 26 16:28:05 2016 From: aledm at qix.co.uk (Aled Morris) Date: Mon, 26 Sep 2016 17:28:05 +0100 Subject: Request for comment -- BCP38 In-Reply-To: References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> Message-ID: On 26 September 2016 at 16:47, Laszlo Hanyecz wrote: > > On 2016-09-26 15:12, Hugo Slabbert wrote: > >> >> If you have links from both ISP A and ISP B and decide to send traffic >> out ISP A's link sourced from addresses ISP B allocated to you, ISP A >> *should* drop that traffic on the floor. > > > This is a legitimate and interesting use case that is broken by BCP38. I don't agree that this is legitimate. Also we're talking about typical mom & pop home users here. I'll sell you a multihoming capable service at a price that includes my time in maintaining your bespoke configuration, but my off-the-shelf home-user service is going to be BCP38. Aled From nanog at ics-il.net Mon Sep 26 16:29:42 2016 From: nanog at ics-il.net (Mike Hammett) Date: Mon, 26 Sep 2016 11:29:42 -0500 (CDT) Subject: Request for comment -- BCP38 In-Reply-To: <20160926162155.GI16464@bamboo.slabnet.com> References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> <20160926162155.GI16464@bamboo.slabnet.com> Message-ID: <1626690466.1943.1474907382437.JavaMail.mhammett@ThunderFuck> I would assume that on a broadband grade connection it shouldn't work unless you have a niche player and proper LOA. I would assume that on a BGP level circuit that it would work, again, given proper documentation (LOAs, IRRDB entry, etc.). IRRDBs make this wonderfully easier. By default, deny. Allow whatever is in the IRRDB entry. $250 for manual changes. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Hugo Slabbert" To: "Mike Hammett" Cc: "John Levine" , nanog at nanog.org Sent: Monday, September 26, 2016 11:21:55 AM Subject: Re: Request for comment -- BCP38 On Mon 2016-Sep-26 11:15:11 -0500, Mike Hammett wrote: >> >>----- Original Message ----- >> >>From: "John Levine" >>To: nanog at nanog.org >>Sent: Monday, September 26, 2016 11:04:33 AM >>Subject: Re: Request for comment -- BCP38 >> >>>If you have links from both ISP A and ISP B and decide to send traffic out >>>ISP A's link sourced from addresses ISP B allocated to you, ISP A *should* >>>drop that traffic on the floor. There is no automated or scalable way for >>>ISP A to distinguish this "legitimate" use from spoofing; unless you >>>consider it scalable for ISP A to maintain thousands if not more >>>"exception" ACLs to uRPF and BCP38 egress filters to cover all of the cases >>>of customers X, Y, and Z sourcing traffic into ISP A's network using IPs >>>allocated to them by other ISPs? >> >>I gather the usual customer response to this is "if you don't want our >>$50K/mo, I'm sure we can find another ISP who does." >> >>From the conversations I've had with ISPs, the inability to manage >>legitimate traffic from dual homed customer networks is the most >>significant bar to widespread BCP38. I realize there's no way to do >>it automatically now, but it doesn't seem like total rocket science to >>come up with some way for providers to pass down a signed object to >>the customer routers that the routers can then pass back up to the >>customer's other providers. >> >>R's, >>John >> >>PS: "Illegitimate" is not a synonym for inconvenient, or hard to handle. >> >Are you talking BGP level customers or individual small businesses' >broadband service? I myself am talking about the latter and included the option of PI space to cover that (although I guess at some point this can be made fly with PA space from another provider if both providers are willing enough to play ball), though from the $50/mo figure John listed, I'm assuming he's talking about the latter. Do people really expect to be able to do this on residential or small business broadband networks? I can't remember any time in recent memory where I assumed I could set a source address to any IP I fancy and have that packet successfully make its way through the SP's network. > >----- >Mike Hammett >Intelligent Computing Solutions >http://www.ics-il.com > >Midwest-IX >http://www.midwest-ix.com -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal From hugo at slabnet.com Mon Sep 26 16:37:05 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 09:37:05 -0700 Subject: Request for comment -- BCP38 In-Reply-To: References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> <20160926162155.GI16464@bamboo.slabnet.com> Message-ID: <20160926163705.GE3770@bamboo.slabnet.com> On Mon 2016-Sep-26 12:25:58 -0400, John R. Levine wrote: >>>>I gather the usual customer response to this is "if you don't want our >>>>$50K/mo, I'm sure we can find another ISP who does." > >>I myself am talking about the latter and included the option of PI >>space to cover that (although I guess at some point this can be >>made fly with PA space from another provider if both providers are >>willing enough to play ball), though from the $50/mo figure John >>listed, I'm assuming he's talking about the latter. > >Who said $50/mo? Apparently I need even more caffeine that I first imagined... If we're talking about networks with that kind of MRC, is it really that far of a stretch to require PI space for this? Then again: If we're talking about that kind of MRC, then I'm assuming ISP A can be coaxed to allow explicit and well-defined exceptions on the customer's links. This discussion started wrt to COTS dual-ISP routers though, as mentioned in Ken Chase's message, no? Where I'm assuming we're talking about mom-and-pop operations rather than a $50K/mo business account. > >Regards, >John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", >Please consider the environment before reading this e-mail. https://jl.ly -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From johnl at iecc.com Mon Sep 26 16:39:21 2016 From: johnl at iecc.com (John R. Levine) Date: 26 Sep 2016 12:39:21 -0400 Subject: Request for comment -- BCP38 In-Reply-To: <20160926163705.GE3770@bamboo.slabnet.com> References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> <20160926162155.GI16464@bamboo.slabnet.com> <20160926163705.GE3770@bamboo.slabnet.com> Message-ID: > If we're talking about networks with that kind of MRC, is it really that far > of a stretch to require PI space for this? Then again: If we're talking > about that kind of MRC, then I'm assuming ISP A can be coaxed to allow > explicit and well-defined exceptions on the customer's links. Yes. A) Check the prices for PI space, if you can even get any. B) Our network works fine with PA space. If you claim otherwise, we know plenty of other operators who aren't as wedged as you are. Trying to solve technical problems with social solutions rarely works. See for example, the sad history of passwords. Regards, John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly From hugo at slabnet.com Mon Sep 26 16:44:06 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 09:44:06 -0700 Subject: Request for comment -- BCP38 In-Reply-To: References: <20160926160433.14119.qmail@ary.lan> <303915647.1728.1474906508724.JavaMail.mhammett@ThunderFuck> <20160926162155.GI16464@bamboo.slabnet.com> <20160926163705.GE3770@bamboo.slabnet.com> Message-ID: <20160926164406.GF3770@bamboo.slabnet.com> On Mon 2016-Sep-26 12:39:21 -0400, John R. Levine wrote: >>If we're talking about networks with that kind of MRC, is it really >>that far of a stretch to require PI space for this? Then again: >>If we're talking about that kind of MRC, then I'm assuming ISP A >>can be coaxed to allow explicit and well-defined exceptions on the >>customer's links. > >Yes. > >A) Check the prices for PI space, if you can even get any. > >B) Our network works fine with PA space. If you claim otherwise, we >know plenty of other operators who aren't as wedged as you are. > >Trying to solve technical problems with social solutions rarely >works. See for example, the sad history of passwords. Mike and Aled covering this better than I am. > >Regards, >John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", >Please consider the environment before reading this e-mail. https://jl.ly -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From johnl at iecc.com Mon Sep 26 18:03:55 2016 From: johnl at iecc.com (John Levine) Date: 26 Sep 2016 18:03:55 -0000 Subject: Request for comment -- BCP38 In-Reply-To: Message-ID: <20160926180355.1229.qmail@ary.lan> >>> >>> If you have links from both ISP A and ISP B and decide to send traffic >>> out ISP A's link sourced from addresses ISP B allocated to you, ISP A >>> *should* drop that traffic on the floor. > >> This is a legitimate and interesting use case that is broken by BCP38. > >I don't agree that this is legitimate. > >Also we're talking about typical mom & pop home users here. There are SOHO modems that will fall back to a second connection if the primary one fails, but that's not what we're talking about here. The customers I'm talking about are businesses large enough to have two dedicated upstreams, and a chunk of address spaced SWIP'ed from each. Some run BGP but I get the impression as likely as not they have static routes to the two upstreams. For people who missed it the last time, I said $50K/mo, not $50/mo. Letters matter. R's, John From laszlo at heliacal.net Mon Sep 26 18:44:41 2016 From: laszlo at heliacal.net (Laszlo Hanyecz) Date: Mon, 26 Sep 2016 18:44:41 +0000 Subject: Request for comment -- BCP38 In-Reply-To: <20160926180355.1229.qmail@ary.lan> References: <20160926180355.1229.qmail@ary.lan> Message-ID: On 2016-09-26 18:03, John Levine wrote: >>>> If you have links from both ISP A and ISP B and decide to send traffic >>>> out ISP A's link sourced from addresses ISP B allocated to you, ISP A >>>> *should* drop that traffic on the floor. >>> This is a legitimate and interesting use case that is broken by BCP38. >> I don't agree that this is legitimate. >> >> Also we're talking about typical mom & pop home users here. > There are SOHO modems that will fall back to a second connection if > the primary one fails, but that's not what we're talking about here. > > The customers I'm talking about are businesses large enough to have > two dedicated upstreams, and a chunk of address spaced SWIP'ed from > each. Some run BGP but I get the impression as likely as not they > have static routes to the two upstreams. > > For people who missed it the last time, I said $50K/mo, not $50/mo. Letters matter. This doesn't have to be $50k/mo though. If the connections weren't source address filtered for BCP38 and you could send packets down either one, the CPE could simply start with 2 default routes and take one out when it sees a connection go down. This could work with a cable + DSL connection even. It would be easy to further refine which connection to use for a particular service by simply adding a specific route for that service's address. This would be a lot better than having to restart everything after one of the connections fails. This would provide functionality similar to the BGP setup without any additional work from the service provider. People can't build CPE software that does this type of connection balancing because they can't rely on this working due to BCP38 implementation. In my experience the only way you can get people to stop source address filtering is if you mention BGP, but BGP shouldn't be required to do this. -Laszlo > > R's, > John > From nellermann at broadaspect.com Mon Sep 26 18:58:30 2016 From: nellermann at broadaspect.com (Nick Ellermann) Date: Mon, 26 Sep 2016 18:58:30 +0000 Subject: Lightower (ASN:46887) RTBH community info In-Reply-To: References: Message-ID: I would love to know which ISP's support RTBH? We have struggled to get anyone to support communities. If their website says they do, their NOC sure doesn't know it or agree... I don't want to list names here, but truly we would love RTBH capabilities with our upstreams. Sincerely, Nick Ellermann ? CTO & VP Cloud Services BroadAspect ? E: nellermann at broadaspect.com P: 703-297-4639 F: 703-996-4443 ? THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Matthew Crocker Sent: Thursday, September 22, 2016 1:33 PM To: NANOG Subject: Lightower (ASN:46887) RTBH community info Hello, Does anyone know the RTBH community for Lightower? I?ve tried 46887:666 but that doesn?t work. I have a /32 I need to blackhole, Lightower is the last ISP and it doesn?t appear they support RTBH ? Thanks -Matt -- Matthew Crocker President ? Crocker Communications matthew at corp.crocker.com From baldur.norddahl at gmail.com Mon Sep 26 19:05:28 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Mon, 26 Sep 2016 21:05:28 +0200 Subject: Request for comment -- BCP38 In-Reply-To: <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> Message-ID: Den 26. sep. 2016 18.02 skrev "Mike Hammett" : > > The only asymmetric routing broken is when the source isn't in public Internet route-able space. That just leaves those multi-ISP WAN routers that NAT it. Some of our IP transits implement filtering. All of our transits assigned /30 subnets on the transit ports from their own range (the alternate would have be to ask us to supply the /30 from our pool). Our provider edge router will send back ICMP packets using the interface address from the interface that received the original packet. It will then route the packet using our normal routing table. This means we can receive some packet on transit port A and then route out a ICMP response on port B using the interface address from port A. But transit B filters this ICMP packet because it has a source address belonging to transit A. >From this follows that BCP38 can break things like traceroute and path MTU discovery in what is a very common setup. The only reason we do not have a bigger problem is that few networks will have a downward MTU change at this point in the network. Regards Baldur From outsider at scarynet.org Mon Sep 26 19:14:56 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Mon, 26 Sep 2016 21:14:56 +0200 Subject: IP addresses being attacked in Krebs DDoS? Message-ID: Just give me thise ips so i can add em in dronebl Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Brett Glass Datum: 25-09-16 22:01 (GMT+01:00) Aan: NANOG Onderwerp: IP addresses being attacked in Krebs DDoS? As an ISP who is pro-active when it comes to security, I'd like to know what IP address(es) are being hit by the Krebs on Security DDoS attack. If we know, we can warn customers that they are harboring infected PCs and/or IoT devices. (And if all ISPs did this, it would be possible to curtail such attacks and plug the security holes that make them possible.) --Brett Glass, LARIAT.NET From fw at deneb.enyo.de Mon Sep 26 19:22:45 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Mon, 26 Sep 2016 21:22:45 +0200 Subject: Request for comment -- BCP38 In-Reply-To: (Baldur Norddahl's message of "Mon, 26 Sep 2016 21:05:28 +0200") References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> Message-ID: <871t06sby2.fsf@mid.deneb.enyo.de> * Baldur Norddahl: > Den 26. sep. 2016 18.02 skrev "Mike Hammett" : >> >> The only asymmetric routing broken is when the source isn't in public > Internet route-able space. That just leaves those multi-ISP WAN routers > that NAT it. > > Some of our IP transits implement filtering. All of our transits assigned > /30 subnets on the transit ports from their own range (the alternate would > have be to ask us to supply the /30 from our pool). > > Our provider edge router will send back ICMP packets using the interface > address from the interface that received the original packet. It will then > route the packet using our normal routing table. > > This means we can receive some packet on transit port A and then route out > a ICMP response on port B using the interface address from port A. But > transit B filters this ICMP packet because it has a source address > belonging to transit A. Interesting. But this looks like a feature request for the router vendor, and not like an issue with BCP 38 filtering as such. > From this follows that BCP38 can break things like traceroute and path MTU > discovery in what is a very common setup. That doesn't follow. In order to break PMTUD, you also need an MTU drop. Is that a common configuration for routers in points in the network where this would matter? From george at cbcast.com Mon Sep 26 19:23:10 2016 From: george at cbcast.com (George Skorup) Date: Mon, 26 Sep 2016 14:23:10 -0500 Subject: Lightower (ASN:46887) RTBH community info In-Reply-To: References: Message-ID: <707add15-3fb3-408d-5532-623393097fa1@cbcast.com> Windstream: 7029:4507 GTT: 3257:2666 are the two I know of. On 9/26/2016 1:58 PM, Nick Ellermann wrote: > I would love to know which ISP's support RTBH? We have struggled to get anyone to support communities. If their website says they do, their NOC sure doesn't know it or agree... I don't want to list names here, but truly we would love RTBH capabilities with our upstreams. > > > Sincerely, > Nick Ellermann ? CTO & VP Cloud Services > BroadAspect > > E: nellermann at broadaspect.com > P: 703-297-4639 > F: 703-996-4443 > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. > > > -----Original Message----- > From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Matthew Crocker > Sent: Thursday, September 22, 2016 1:33 PM > To: NANOG > Subject: Lightower (ASN:46887) RTBH community info > > Hello, > > Does anyone know the RTBH community for Lightower? I?ve tried 46887:666 but that doesn?t work. I have a /32 I need to blackhole, Lightower is the last ISP and it doesn?t appear they support RTBH ? > > Thanks > > -Matt > > -- > Matthew Crocker > President ? Crocker Communications > matthew at corp.crocker.com > From hugo at slabnet.com Mon Sep 26 19:35:15 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 12:35:15 -0700 Subject: Lightower (ASN:46887) RTBH community info In-Reply-To: <707add15-3fb3-408d-5532-623393097fa1@cbcast.com> References: <707add15-3fb3-408d-5532-623393097fa1@cbcast.com> Message-ID: <20160926193515.GJ16464@bamboo.slabnet.com> Backwards from what you want (sorted by provider rather than supported "feature" e.g. RTBH), but: http://onesc.net/communities/ fwiw -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal On Mon 2016-Sep-26 14:23:10 -0500, George Skorup wrote: >Windstream: 7029:4507 >GTT: 3257:2666 > >are the two I know of. > >On 9/26/2016 1:58 PM, Nick Ellermann wrote: >>I would love to know which ISP's support RTBH? We have struggled to get anyone to support communities. If their website says they do, their NOC sure doesn't know it or agree... I don't want to list names here, but truly we would love RTBH capabilities with our upstreams. >> >> >>Sincerely, >>Nick Ellermann ? CTO & VP Cloud Services >>BroadAspect >>E: nellermann at broadaspect.com >>P: 703-297-4639 >>F: 703-996-4443 >>THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. >> >> >>-----Original Message----- >>From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Matthew Crocker >>Sent: Thursday, September 22, 2016 1:33 PM >>To: NANOG >>Subject: Lightower (ASN:46887) RTBH community info >> >>Hello, >> >>Does anyone know the RTBH community for Lightower? I?ve tried 46887:666 but that doesn?t work. I have a /32 I need to blackhole, Lightower is the last ISP and it doesn?t appear they support RTBH ? >> >>Thanks >> >>-Matt >> >>-- >>Matthew Crocker >>President ? Crocker Communications >>matthew at corp.crocker.com >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: From baldur.norddahl at gmail.com Mon Sep 26 20:08:36 2016 From: baldur.norddahl at gmail.com (Baldur Norddahl) Date: Mon, 26 Sep 2016 22:08:36 +0200 Subject: Request for comment -- BCP38 In-Reply-To: <871t06sby2.fsf@mid.deneb.enyo.de> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> Message-ID: <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> This means we can receive some packet on transit port A and then route out >> a ICMP response on port B using the interface address from port A. But >> transit B filters this ICMP packet because it has a source address >> belonging to transit A. > Interesting. But this looks like a feature request for the router > vendor, and not like an issue with BCP 38 filtering as such. Can you quote an RFC for anything that the router is doing wrong? Is there a requirement that a router must support source routing? In our case we actually did contact the vendor. Turns out that it will do source routing but not for packets from the control plane. There is no way to resolve the issue with the current software available to us. The vendor is not priotizing fixing this as I am also unable to point to any RFC that is being violated. Even if or when we get a fix, this will continue to be a problem because it is a very common setup. There are thousands of networks out there that have the issue and many of them are probably not even realising it. > >> From this follows that BCP38 can break things like traceroute and path MTU >> discovery in what is a very common setup. > That doesn't follow. In order to break PMTUD, you also need an MTU > drop. Is that a common configuration for routers in points in the > network where this would matter? It is not a common configuration which was what I said in the text you removed from that quote: "From this follows that BCP38 can break things like traceroute and path MTU discovery in what is a very common setup. The only reason we do not have a bigger problem is that few networks will have a downward MTU change at this point in the network". Besides it appears that path MTU is broken in general. We accept the problem because the only actual consequence is that traceroute is a little funky. Regards, Baldur From thomas.king at de-cix.net Mon Sep 26 20:08:54 2016 From: thomas.king at de-cix.net (Thomas King) Date: Mon, 26 Sep 2016 20:08:54 +0000 Subject: Lightower (ASN:46887) RTBH community info In-Reply-To: <707add15-3fb3-408d-5532-623393097fa1@cbcast.com> References: <707add15-3fb3-408d-5532-623393097fa1@cbcast.com> Message-ID: <7FFC68F8-8239-47B0-BF32-7B944D018708@de-cix.net> Hi all, there is a well-known BLACKHOLE BGP community underway [0]: 65535:666 All transit providers and IXPs are kindly asked to support it. DE-CIX will support this community on all its IXPs soon. Best regards, Thomas [0]: https://datatracker.ietf.org/doc/draft-ietf-grow-blackholing/ On 26/09/16 21:23, "NANOG on behalf of George Skorup" wrote: Windstream: 7029:4507 GTT: 3257:2666 are the two I know of. On 9/26/2016 1:58 PM, Nick Ellermann wrote: > I would love to know which ISP's support RTBH? We have struggled to get anyone to support communities. If their website says they do, their NOC sure doesn't know it or agree... I don't want to list names here, but truly we would love RTBH capabilities with our upstreams. > > > Sincerely, > Nick Ellermann ? CTO & VP Cloud Services > BroadAspect > > E: nellermann at broadaspect.com > P: 703-297-4639 > F: 703-996-4443 > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. > > > -----Original Message----- > From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Matthew Crocker > Sent: Thursday, September 22, 2016 1:33 PM > To: NANOG > Subject: Lightower (ASN:46887) RTBH community info > > Hello, > > Does anyone know the RTBH community for Lightower? I?ve tried 46887:666 but that doesn?t work. I have a /32 I need to blackhole, Lightower is the last ISP and it doesn?t appear they support RTBH ? > > Thanks > > -Matt > > -- > Matthew Crocker > President ? Crocker Communications > matthew at corp.crocker.com > From lear at cisco.com Mon Sep 26 20:16:41 2016 From: lear at cisco.com (Eliot Lear) Date: Mon, 26 Sep 2016 22:16:41 +0200 Subject: Request for comment -- BCP38 In-Reply-To: References: <20160926180355.1229.qmail@ary.lan> Message-ID: Guys, You're getting wrapped around the axle. Start by solving the 90% problem, and worry about the 10% one later. BGP38 addresses the former very well, and the other 10% requires enough manual labor already that you can charge it back. Eliot On 9/26/16 8:44 PM, Laszlo Hanyecz wrote: > > > On 2016-09-26 18:03, John Levine wrote: >>>>> If you have links from both ISP A and ISP B and decide to send >>>>> traffic >>>>> out ISP A's link sourced from addresses ISP B allocated to you, ISP A >>>>> *should* drop that traffic on the floor. >>>> This is a legitimate and interesting use case that is broken by BCP38. >>> I don't agree that this is legitimate. >>> >>> Also we're talking about typical mom & pop home users here. >> There are SOHO modems that will fall back to a second connection if >> the primary one fails, but that's not what we're talking about here. >> >> The customers I'm talking about are businesses large enough to have >> two dedicated upstreams, and a chunk of address spaced SWIP'ed from >> each. Some run BGP but I get the impression as likely as not they >> have static routes to the two upstreams. >> >> For people who missed it the last time, I said $50K/mo, not $50/mo. >> Letters matter. > > This doesn't have to be $50k/mo though. If the connections weren't > source address filtered for BCP38 and you could send packets down > either one, the CPE could simply start with 2 default routes and take > one out when it sees a connection go down. This could work with a > cable + DSL connection even. It would be easy to further refine which > connection to use for a particular service by simply adding a specific > route for that service's address. This would be a lot better than > having to restart everything after one of the connections fails. > This would provide functionality similar to the BGP setup without any > additional work from the service provider. People can't build CPE > software that does this type of connection balancing because they > can't rely on this working due to BCP38 implementation. In my > experience the only way you can get people to stop source address > filtering is if you mention BGP, but BGP shouldn't be required to do > this. > > -Laszlo > >> >> R's, >> John >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From marka at isc.org Mon Sep 26 21:08:27 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 07:08:27 +1000 Subject: Request for comment -- BCP38 In-Reply-To: Your message of "Mon, 26 Sep 2016 22:16:41 +0200." References: <20160926180355.1229.qmail@ary.lan> Message-ID: <20160926210827.AD092551412D@rock.dv.isc.org> BCP 38 does NOT prevent multi-homed clients. Naive deployment of BCP 38 prevents multi-homed clients. There is NOTHING in BCP 38 that says you may not also accept other prefixes allocated to your clients. BCP 38 says accept allocated address, drop everything else. Now it should be possible with ROA's to completely automate support for multi-homed clients as you can cryptographically verify the addresses allocated to your clients from other ISPs / RIRs. The DHCP server could generate a CERT for every allocation it hands out if a client requested it. This really only needs a DHCP option to be defined to request this. Just as it is possible to secure BGP it is possible to secure BCP 38 additional prefixes. BCP 38 is INGRESS filtering from your customers. Treating your own offices as a "customer" is also recommended. In message , Eliot Lear writes: > This is an OpenPGP/MIME signed message (RFC 4880 and 3156) > --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN > From: Eliot Lear > To: Laszlo Hanyecz , nanog at nanog.org > Message-ID: > Subject: Re: Request for comment -- BCP38 > References: <20160926180355.1229.qmail at ary.lan> > > In-Reply-To: > Content-Type: text/plain; charset=utf-8 > Content-Transfer-Encoding: quoted-printable > > Guys, > > You're getting wrapped around the axle. Start by solving the 90% > problem, and worry about the 10% one later. BGP38 addresses the former > very well, and the other 10% requires enough manual labor already that > you can charge it back. > > Eliot > > > > > On 9/26/16 8:44 PM, Laszlo Hanyecz wrote: > > > > > > On 2016-09-26 18:03, John Levine wrote: > >>>>> If you have links from both ISP A and ISP B and decide to send > >>>>> traffic > >>>>> out ISP A's link sourced from addresses ISP B allocated to you, ISP= > A > >>>>> *should* drop that traffic on the floor. > >>>> This is a legitimate and interesting use case that is broken by BCP3= > 8. > >>> I don't agree that this is legitimate. > >>> > >>> Also we're talking about typical mom & pop home users here. > >> There are SOHO modems that will fall back to a second connection if > >> the primary one fails, but that's not what we're talking about here. > >> > >> The customers I'm talking about are businesses large enough to have > >> two dedicated upstreams, and a chunk of address spaced SWIP'ed from > >> each. Some run BGP but I get the impression as likely as not they > >> have static routes to the two upstreams. > >> > >> For people who missed it the last time, I said $50K/mo, not $50/mo.=20 > >> Letters matter. > > > > This doesn't have to be $50k/mo though. If the connections weren't > > source address filtered for BCP38 and you could send packets down > > either one, the CPE could simply start with 2 default routes and take > > one out when it sees a connection go down. This could work with a > > cable + DSL connection even. It would be easy to further refine which > > connection to use for a particular service by simply adding a specific > > route for that service's address. This would be a lot better than > > having to restart everything after one of the connections fails. =20 > > This would provide functionality similar to the BGP setup without any > > additional work from the service provider. People can't build CPE > > software that does this type of connection balancing because they > > can't rely on this working due to BCP38 implementation. In my > > experience the only way you can get people to stop source address > > filtering is if you mention BGP, but BGP shouldn't be required to do > > this. > > > > -Laszlo > > > >> > >> R's, > >> John > >> > > > > > > > > --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN > Content-Type: application/pgp-signature; name="signature.asc" > Content-Description: OpenPGP digital signature > Content-Disposition: attachment; filename="signature.asc" > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2 > > iQEcBAEBCAAGBQJX6YIpAAoJEIe2a0bZ0nozGHAH/3EXMt2/t68dgB92RPYyVSHR > 46WpPPOEhD1Qd1s6MW4mhzhWWmv99RGwPm1ZXVzE9iIpMkptXEuzbsHz1V7mjsao > 5MkIyzvWgF6qpe1kI1xZp2xoDa++XjfJqwUMg0swxji7idjkMILw6buE70lubOCB > Uu2Y6uGPCnsIEcD106AxJYkP91BlBkBRPAFoBjbdZKRTs3+TYQgxS815qviSiDux > MXhdxgpo8yg/B8knmjQwwIcG3+Ug5FvkPJyz88GQKwwU6nEIKUn2Zf3U/vw2vQTN > 3K+DlRIGy6ZXani4ab58pswrrhFl9P9bocRom+cJQAqb3JwR60NuUX1/YKeTv2o= > =HdDL > -----END PGP SIGNATURE----- > > --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN-- -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From list at satchell.net Mon Sep 26 21:13:39 2016 From: list at satchell.net (Stephen Satchell) Date: Mon, 26 Sep 2016 14:13:39 -0700 Subject: is someone with BCP38.info update privs summarizing this discussion? Message-ID: I think some pretty good information has surfaced, that would be WONDERFUL to have on that site. From nellermann at broadaspect.com Mon Sep 26 21:27:25 2016 From: nellermann at broadaspect.com (Nick Ellermann) Date: Mon, 26 Sep 2016 21:27:25 +0000 Subject: routing issues between NTT/ATT/Level3? Message-ID: Starting on 9/23 and really bad over the weekend we have had issues with end points out on AT&T's US network specifically in the Texas and Michigan areas. When our end points with AT&T drop, we end up losing monitored endpoints on Level3's network as well. Really bad periods of packetloss. Both of which for us are utilizing NTT as our upstream out of Ashburn. We have opened a ticket with NTT. Anyone else seeing similar issues? Since about noon today 9/26, it's gotten a little better but still periods of packet loss and dropped customer VPN tunnels. Probably not bothering the average web surfer but killing our customers voip and vpn traffic. It's really hard to diagnose and fix other vendors' networks! Sincerely, Nick Ellermann - CTO & VP Cloud Services BroadAspect E: nellermann at broadaspect.com P: 703-297-4639 F: 703-996-4443 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. From marka at isc.org Mon Sep 26 23:09:46 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 09:09:46 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "26 Sep 2016 15:56:49 +0000." <20160926155649.14061.qmail@ary.lan> References: <20160926155649.14061.qmail@ary.lan> Message-ID: <20160926230946.685605514EDF@rock.dv.isc.org> In message <20160926155649.14061.qmail at ary.lan>, "John Levine" writes: > >>That paper is about reflection attacks. From what I've read, this was > >>not a reflection attack. The IoT devices are infected with botware > >>which sends attack traffic directly. Address spoofing is not particularly > >>useful for controlling botnets. > > > >But that's not only remaining use of source address spoofing in direct > >attacks, no? Even if reflection and amplification are not used, spoofing > >can still be used for obfuscation. > > I agree that it would be nice if more networks did ingress filtering, > but if you're expecting a major decrease in evil, you will be > disappointed. > > At this point it's mostly useful for identifying the guilty or > negligent parties afterwards. > > R's, > John Actually for ISP's that do it, it can becomes a source of intelligence on where the compromised / misconfigured machines are. A good ISP would be informing their customers that they are seeing anomalous traffic. With IPv6 there is likely to be more of this traffic visible as home NATs wont be masking it. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From Jason_Livingood at comcast.com Mon Sep 26 23:28:13 2016 From: Jason_Livingood at comcast.com (Livingood, Jason) Date: Mon, 26 Sep 2016 23:28:13 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> References: <20160924144757.6291.qmail@ary.lan> <20160924184332.GA45065@excession.tpb.net> <6CB7A90CA0C0D3CE.6E724E09-9FD0-4486-89BF-1A624A0F076A@mail.outlook.com> <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> Message-ID: On 9/25/16, 5:57 PM, "NANOG on behalf of Patrick W. Gilmore" wrote: > Yeah, ?cause that was so successful in the past. > Remember University of Wisconsin vs. D-Link and their hard-coded NTP server address? Ha! Yeah, an oldie but a goodie. Anyway, maybe this time will be different? (I?m an optimist.) FWIW, a few of the list members here are working on a BITAG paper on this ? which will likely get some traction in policy/regulatory circles once completed. A simple paper certainly won?t turn the tide but if the paper is finished soon it presents an opportunity to get a bit wider notice & impact than it perhaps otherwise would. Jason From Jason_Livingood at comcast.com Mon Sep 26 23:32:12 2016 From: Jason_Livingood at comcast.com (Livingood, Jason) Date: Mon, 26 Sep 2016 23:32:12 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160926230946.685605514EDF@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> Message-ID: <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews" wrote: > A good ISP would be informing their customers that they are seeing anomalous traffic. Therein lies the problem if the traffic does not look anomalous I suppose. But even if it does look unusual, ISPs would be asking consumers to trash/update/turn off a lot of devices in time ? like when every home has 10s or 100s of these devices. ISP: Dear customer, looks like one of your light switches is sending spam. Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and? ;-) Jason From marka at isc.org Mon Sep 26 23:41:42 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 09:41:42 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Mon, 26 Sep 2016 23:32:12 +0000." <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> Message-ID: <20160926234142.6E7705515473@rock.dv.isc.org> In message <03DC1038-024A-4D9F-AC5B-3E88CDF56246 at cable.comcast.com>, "Livingood, Jason" writes: > On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews" marka at isc.org> wrote: > > A good ISP would be informing their customers that they are seeing > anomalous traffic. > > Therein lies the problem if the traffic does not look anomalous I > suppose. But even if it does look unusual, ISPs would be asking consumers > to trash/update/turn off a lot of devices in time like when every home > has 10s or 100s of these devices. > ISP: Dear customer, looks like one of your light switches is sending spam. > Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 > smart TVs, and 3 smart thermostats, and 6 cameras, and > > ;-) > > Jason > Dear customer, we are seeing traffic coming from your network. If you need help isolating the source of the traffic here are a few companies in your city that can help you. This is not a exhaustive list. Support -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From marka at isc.org Mon Sep 26 23:49:39 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 09:49:39 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Tue, 27 Sep 2016 09:41:42 +1000." <20160926234142.6E7705515473@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> Message-ID: <20160926234939.B1961551553A@rock.dv.isc.org> In message <20160926234142.6E7705515473 at rock.dv.isc.org>, Mark Andrews writes: > > In message <03DC1038-024A-4D9F-AC5B-3E88CDF56246 at cable.comcast.com>, "Livingood, Jason" writes: > > On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews" > marka at isc.org> wrote: > > > A good ISP would be informing their customers that they are seeing > > anomalous traffic. > > > > Therein lies the problem if the traffic does not look anomalous I > > suppose. But even if it does look unusual, ISPs would be asking consumers > > to trash/update/turn off a lot of devices in time like when every home > > has 10s or 100s of these devices. > > ISP: Dear customer, looks like one of your light switches is sending spam. > > Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 > > smart TVs, and 3 smart thermostats, and 6 cameras, and > > > > ;-) > > > > Jason > > > > Dear customer, > we are seeing traffic coming from your network. > > If you need help isolating the source of the traffic here are a few > companies in your city that can help you. > > > > This is not a exhaustive list. > > Support > > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka at isc.org Giving them real time access to the anomalous traffic log feed for their residence would also help. They or the specialist they bring in will be able to use that to trace back the problem. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From morrowc.lists at gmail.com Mon Sep 26 23:58:51 2016 From: morrowc.lists at gmail.com (Christopher Morrow) Date: Mon, 26 Sep 2016 19:58:51 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160926234939.B1961551553A@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews wrote: > > Giving them real time access to the anomalous traffic log feed for > their residence would also help. They or the specialist they bring > in will be able to use that to trace back the problem. > > wouldn't this work better as a standard bit of CPE software capability? wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE and kept for ~30mins (just guessing) in a circular buffer be 'good enough' to present a pretty clear UI to the user? ip/mac/vendor sending (webtraffic|email|probes) to destination-name [checkbox] select those youd' like to block [clickhere] This really doesn't seem hard, to present in a fairly straight forward manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something similar to this approach... but on the other hand: "At least my ISP isn't snooping on all my traffic" From johnl at iecc.com Tue Sep 27 00:13:02 2016 From: johnl at iecc.com (John R. Levine) Date: 26 Sep 2016 20:13:02 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> Message-ID: > Therein lies the problem if the traffic does not look anomalous I suppose. But even if it does look unusual, ISPs would be asking consumers to trash/update/turn off a lot of devices in time ? like when every home has 10s or 100s of these devices. > ISP: Dear customer, looks like one of your light switches is sending spam. > Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and? That's why turning them off has to be mandatory if the ISP can't mitigate the traffic in real time. Sorry, but something in your house is attacking strangers. Once you figure out what it is, here's a handy list of links to the ongoing class action suits against the manufacturers. Regards, John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly From marka at isc.org Tue Sep 27 00:19:24 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 10:19:24 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Mon, 26 Sep 2016 19:58:51 -0400." References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: <20160927001924.B7DE255159E3@rock.dv.isc.org> In message , Christopher Morrow writes: > On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews wrote: > > > > > Giving them real time access to the anomalous traffic log feed for > > their residence would also help. They or the specialist they bring > > in will be able to use that to trace back the problem. > > > > > wouldn't this work better as a standard bit of CPE software capability? While this would be useful, it is not sufficient. This may or may not match what the ISP is detecting and basing its reports to the customer on. Such a feed could also include MTA logs for email nominally from the customer. etc. If we want customers to clean up networks, use of the ISP's services, they need to be able to see what is going wrong as far as the ISP is concerned. You guys complain when you don't get good data to chase down abuse reports. Your customers need the same data. The more you can automate this the cheaper it is to provide. Additional the more you inform your customers, the better they will feel about you as a ISP, and the less abuse reports you will get from external sources as a compromised box can do anything. It's in your long term interests to get that compromised box fixed / removed. Yes, there will be setup / development costs. > wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE > and kept for ~30mins (just guessing) in a circular buffer be 'good enough' > to present a pretty clear UI to the user? > > ip/mac/vendor sending (webtraffic|email|probes) to destination-name > [checkbox] > > > > select those youd' like to block [clickhere] > > This really doesn't seem hard, to present in a fairly straight forward > manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something > similar to this approach... but on the other hand: > "At least my ISP isn't snooping on all my traffic" > -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From rdobbins at arbor.net Tue Sep 27 04:05:19 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 11:05:19 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: On 27 Sep 2016, at 6:58, Christopher Morrow wrote: > wouldn't something as simple as netflow/sflow/ipfix synthesized on the > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good > enough' to present a pretty clear UI to the user? +1 for this capability in CPE. OTOH, it will be of no use whatsoever to the user. Providing the user with access to anomalous traffic feeds won't help, either. Users aren't going to call in some third-party service/support company, either. It call comes down to the network operator, one way or another. There's no separation in the public mind of 'my network' from 'the Internet' that is analogous to the separation between 'the power company' and 'the electrical wiring in my house/apartment' (and even in that space, the conceptual separation often isn't present). ----------------------------------- Roland Dobbins From hugo at slabnet.com Tue Sep 27 04:37:42 2016 From: hugo at slabnet.com (Hugo Slabbert) Date: Mon, 26 Sep 2016 21:37:42 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <20160926210827.AD092551412D@rock.dv.isc.org> References: <20160926180355.1229.qmail@ary.lan> <20160926210827.AD092551412D@rock.dv.isc.org> Message-ID: <20160927043742.njtfx7xytjcbflfx@slab-wks-04.int.slabnet.com> This seems to have split into a few different sub-threads and had some cross-talk on which network is being described where (thanks in no small part to my flub on John's figures and target), but this seems to be exactly the kind of info folks are looking for but missing at http://www.bcp38.info. In the interest of clarity, does this roughly cover the options/challenges people are seeing at various types of networks? Network #1 ---------- Customer has PI space. ISP A learns routes to the customer's PI space across the customer links via BGP or even static routes. ISP A's active forwarding path to the PI space is via the customer link. We're assuming an edge provider, i.e. the customer is not providing transit for further downstream networks. BCP 38 implementation: Strict uRPF on customer-facing interfaces should "Just Work". If egress source address filtering is implemented, this can/should be populated from IRR data. Network #2 ---------- Same as #1, except while ISP A does have a valid forwarding path to the customer's PI space via the customer link, that is *not* the active path. Examples could be that customer prefers ingress via another link (either with ISP A or another provider altogether) and so prepends, uses provider communities to depref the advertisement below another path, etc. BCP38 implementation: Feasible Path RPF[1] should work, but AFAIK is not supported on all platforms. Failure modes should be considered, though, and using feasible path RPF would result in dropped traffic if the customer dropped the announcement to ISP A in the future. If Feasible Path RPF not supported or viable, IRR data can/should be used to generate customer interface input filters. Same as #1, if egress source address filtering is implemented, this can/should be populated from IRR data. Network #3 ---------- Same as #1, except the transit provider has *no* valid forwarding path to the customer's PI space via the customer link. IOW the customer is using the transit provider for egress *only*, not ingress. Feasible Path RPF is not viable. Input filters are required on customer-facing interfaces, and can/should be generated from IRR data. Same as #1 and 2, if egress source address filtering is implemented, this can/should be populated from IRR data. PA space variants to Networks #1 - 3 ------------------------------------ Flipping all of the above scenarios to PA space rather than PI, the options for implementing uRPF are the same. Where I would consider things becoming more challenging is on generating input filters on customer-facing interfaces were strict or Feasible Path RPF are not viable, or egress source address filters if those are in use. Mark: You mentioned the option of leveraging ROAs for validating customer prefixes allocated by other ISPs. I must admit my RPKI knowledge needs some love, but my understanding is that ROAs link prefixes to a given ASN, with the ROA signed by the private key from the resource holder's certificate. Would this validation option for PA allocations be applicable only in cases where the customer holds an ASN to which the prefix could be linked? In other words the prefix on the ROA would be the PA prefix allocated by ISP B, the ASN on the ROA would be the customer's ASN, and the private key signing the ROA would belong to ISP B, as they are the resource holder? Hopefully not downgrading this conversation, but lacking RPKI validators at ISP A and a valid RPKI setup at ISP B, and assuming that the customer has an ASN, this info could also be generated from IRR data as well, no? I mean, dropping a /29 into a routing registry won't do much good in terms of getting announcements accepted, but it could still be leveraged for this use case to generate filter prefix lists, right? If this is not tied to a customer ASN (because they don't have one) but rather to ISP B's ASN, how does ISP A vet that a ROA-validated prefix tied to ISP B's ASN applies to this particular customer? Absent RPKI and falling back to IRR, same question? Network #4 ---------- Customer has broadband connections from ISP A and ISP B. If I'm not totally out to lunch above re: needing an ASN for the customer to leverage an ROA-based solution as Mark touched on, are we stuck in either manual ACLs or asking or writing new implementations as John described? On Mon 2016-Sep-26 16:04:33 -0000, John Levine wrote: ... > I realize there's no way to do it automatically now, but it doesn't seem > like total rocket science to come up with some way for providers to pass > down a signed object to the customer routers that the routers can then > pass back up to the customer's other providers. Transit touchdown interfaces ---------------------------- This is Baldur's scenario. Barring both upstreams maintaining manual filters that cover the touchdown networks handed to their mutual customers, it seems either Mark's or John's suggestions could be potential solutions here? -- Hugo Slabbert | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E | also on Signal [1] https://tools.ietf.org/html/rfc3704#section-2.3 On Tue 2016-Sep-27 07:08:27 +1000, Mark Andrews wrote: > >BCP 38 does NOT prevent multi-homed clients. Naive deployment of >BCP 38 prevents multi-homed clients. There is NOTHING in BCP 38 that >says you may not also accept other prefixes allocated to your >clients. > >BCP 38 says accept allocated address, drop everything else. > >Now it should be possible with ROA's to completely automate support >for multi-homed clients as you can cryptographically verify the >addresses allocated to your clients from other ISPs / RIRs. > >The DHCP server could generate a CERT for every allocation it hands >out if a client requested it. This really only needs a DHCP option >to be defined to request this. > >Just as it is possible to secure BGP it is possible to secure BCP 38 >additional prefixes. > >BCP 38 is INGRESS filtering from your customers. Treating your own >offices as a "customer" is also recommended. > >In message , Eliot Lear writes: >> This is an OpenPGP/MIME signed message (RFC 4880 and 3156) >> --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN >> From: Eliot Lear >> To: Laszlo Hanyecz , nanog at nanog.org >> Message-ID: >> Subject: Re: Request for comment -- BCP38 >> References: <20160926180355.1229.qmail at ary.lan> >> >> In-Reply-To: >> Content-Type: text/plain; charset=utf-8 >> Content-Transfer-Encoding: quoted-printable >> >> Guys, >> >> You're getting wrapped around the axle. Start by solving the 90% >> problem, and worry about the 10% one later. BGP38 addresses the former >> very well, and the other 10% requires enough manual labor already that >> you can charge it back. >> >> Eliot >> >> >> >> >> On 9/26/16 8:44 PM, Laszlo Hanyecz wrote: >> > >> > >> > On 2016-09-26 18:03, John Levine wrote: >> >>>>> If you have links from both ISP A and ISP B and decide to send >> >>>>> traffic >> >>>>> out ISP A's link sourced from addresses ISP B allocated to you, ISP= >> A >> >>>>> *should* drop that traffic on the floor. >> >>>> This is a legitimate and interesting use case that is broken by BCP3= >> 8. >> >>> I don't agree that this is legitimate. >> >>> >> >>> Also we're talking about typical mom & pop home users here. >> >> There are SOHO modems that will fall back to a second connection if >> >> the primary one fails, but that's not what we're talking about here. >> >> >> >> The customers I'm talking about are businesses large enough to have >> >> two dedicated upstreams, and a chunk of address spaced SWIP'ed from >> >> each. Some run BGP but I get the impression as likely as not they >> >> have static routes to the two upstreams. >> >> >> >> For people who missed it the last time, I said $50K/mo, not $50/mo.=20 >> >> Letters matter. >> > >> > This doesn't have to be $50k/mo though. If the connections weren't >> > source address filtered for BCP38 and you could send packets down >> > either one, the CPE could simply start with 2 default routes and take >> > one out when it sees a connection go down. This could work with a >> > cable + DSL connection even. It would be easy to further refine which >> > connection to use for a particular service by simply adding a specific >> > route for that service's address. This would be a lot better than >> > having to restart everything after one of the connections fails. =20 >> > This would provide functionality similar to the BGP setup without any >> > additional work from the service provider. People can't build CPE >> > software that does this type of connection balancing because they >> > can't rely on this working due to BCP38 implementation. In my >> > experience the only way you can get people to stop source address >> > filtering is if you mention BGP, but BGP shouldn't be required to do >> > this. >> > >> > -Laszlo >> > >> >> >> >> R's, >> >> John >> >> >> > >> > >> >> >> >> --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN >> Content-Type: application/pgp-signature; name="signature.asc" >> Content-Description: OpenPGP digital signature >> Content-Disposition: attachment; filename="signature.asc" >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG/MacGPG2 v2 >> >> iQEcBAEBCAAGBQJX6YIpAAoJEIe2a0bZ0nozGHAH/3EXMt2/t68dgB92RPYyVSHR >> 46WpPPOEhD1Qd1s6MW4mhzhWWmv99RGwPm1ZXVzE9iIpMkptXEuzbsHz1V7mjsao >> 5MkIyzvWgF6qpe1kI1xZp2xoDa++XjfJqwUMg0swxji7idjkMILw6buE70lubOCB >> Uu2Y6uGPCnsIEcD106AxJYkP91BlBkBRPAFoBjbdZKRTs3+TYQgxS815qviSiDux >> MXhdxgpo8yg/B8knmjQwwIcG3+Ug5FvkPJyz88GQKwwU6nEIKUn2Zf3U/vw2vQTN >> 3K+DlRIGy6ZXani4ab58pswrrhFl9P9bocRom+cJQAqb3JwR60NuUX1/YKeTv2o= >> =HdDL >> -----END PGP SIGNATURE----- >> >> --dWOXeq1OLTVjQNjObs7KPBUAPtMRfKIWN-- >-- >Mark Andrews, ISC >1 Seymour St., Dundas Valley, NSW 2117, Australia >PHONE: +61 2 9871 4742 INTERNET: marka at isc.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From marka at isc.org Tue Sep 27 04:43:36 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 14:43:36 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Tue, 27 Sep 2016 11:05:19 +0700." References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: <20160927044337.3586D5519DB9@rock.dv.isc.org> In message , Roland Dobbins wri tes: > > On 27 Sep 2016, at 6:58, Christopher Morrow wrote: > > > wouldn't something as simple as netflow/sflow/ipfix synthesized on the > > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good > > enough' to present a pretty clear UI to the user? > > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different. > It call comes down to the network operator, one way or another. There's > no separation in the public mind of 'my network' from 'the Internet' > that is analogous to the separation between 'the power company' and 'the > electrical wiring in my house/apartment' (and even in that space, the > conceptual separation often isn't present). Actually I don't believe that. They do know what machines they have have connected to their home network. Boxes don't magically connect. Every machine was explictly connected. Mark > ----------------------------------- > Roland Dobbins -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From rdobbins at arbor.net Tue Sep 27 04:56:04 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 11:56:04 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160927044337.3586D5519DB9@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> Message-ID: On 27 Sep 2016, at 11:43, Mark Andrews wrote: > Why not? You call a washing machine mechanic when the washing machine > plays up. This is not conceptually different. Washing machines aren't a utility. Internet is viewed as a utility. > Actually I don't believe that. They do know what machines they have > have connected to their home network. Boxes don't magically > connect. Every machine was explictly connected. First of all, not every devices was explicitly connected by the user. Think set-top boxes/DVRs. Secondly, users connect things an then don't think about them, don't remember credentials, had a horrible ordeal (from their perspective) connecting said devices and then promptly forgot how to administer them. Thirdly, expecting users to troubleshoot which of their devices is emanating bad traffic is unrealistic. The only effective consumer remediation efforts we've seen to date have been broadband access ISPs proactively scanning their customer networks and contacting them when exploitable devices and compromised PCs have been found. Although it's a lot of work, that kind of thing can be done for CPE broadband routers; it can't be done for the things sitting behind those devices, which are doing NAT/firewalling. The partial exception is PCs, because everyone thinks of those when they think of 'the Internet'. And the fact that even their lightbulbs are being connected now - i.e., the huge proliferation of connected devices - militates against user troubleshooting, as well. ----------------------------------- Roland Dobbins From marka at isc.org Tue Sep 27 05:14:33 2016 From: marka at isc.org (Mark Andrews) Date: Tue, 27 Sep 2016 15:14:33 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Tue, 27 Sep 2016 11:56:04 +0700." References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> Message-ID: <20160927051433.49F85551A30E@rock.dv.isc.org> In message , Roland Dobbins writes: > On 27 Sep 2016, at 11:43, Mark Andrews wrote: > > > Why not? You call a washing machine mechanic when the washing machine > > plays up. This is not conceptually different. > > Washing machines aren't a utility. Internet is viewed as a utility. > > > Actually I don't believe that. They do know what machines they have > > have connected to their home network. Boxes don't magically > > connect. Every machine was explictly connected. > > First of all, not every devices was explicitly connected by the user. > Think set-top boxes/DVRs. I'm yet to see a set top box, DVR, TV, games console, phone, etc. that didn't require selecting the WiFi SSID or require you to plug in a ethernet cable. As I said, they don't magically connect to the network. Someone did something to permit them to connect. > Secondly, users connect things an then don't think about them, don't > remember credentials, had a horrible ordeal (from their perspective) > > Thirdly, expecting users to troubleshoot which of their devices is > emanating bad traffic is unrealistic. Which is why there are computer technitions. If you have a fault with a fan you call a electrian. If you have a problem with a toilet you call a plumber. Why do you think people are incapable of calling in someone to help them fix a known issue. > The only effective consumer remediation efforts we've seen to date have > been broadband access ISPs proactively scanning their customer networks > and contacting them when exploitable devices and compromised PCs have > been found. Although it's a lot of work, that kind of thing can be done > for CPE broadband routers; it can't be done for the things sitting > behind those devices, which are doing NAT/firewalling. The partial > exception is PCs, because everyone thinks of those when they think of > 'the Internet'. > > And the fact that even their lightbulbs are being connected now - i.e., > the huge proliferation of connected devices - militates against user > troubleshooting, as well. > > ----------------------------------- > Roland Dobbins -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From rdobbins at arbor.net Tue Sep 27 05:29:17 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 12:29:17 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160927051433.49F85551A30E@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> <20160927051433.49F85551A30E@rock.dv.isc.org> Message-ID: <0530CC85-A04B-4E57-AED9-108ABCF8C137@arbor.net> On 27 Sep 2016, at 12:14, Mark Andrews wrote: > I'm yet to see a set top box, DVR, TV, games console, phone, etc. that > didn't require selecting the WiFi SSID or require you to plug > in a ethernet cable. I've 'seen' tens of millions of them, worldwide. You're generalizing your particular connection/personal provisioning model. > As I said, they don't magically connect to the network. Someone did > something to permit them to connect. That someone quite often isn't the end-user. And as noted previously in this thread, even when users themselves do this, they promptly forget how they did it, lose the documentation, etc. > Why do you think people are incapable of calling in someone to help > them fix a known issue. 1. Because they demonstrably don't. 2. Because it's not perceived as a 'computer problem' - it's perceived as an 'Internet problem', and the 'Internet technician' = the broadband access operator's help-desk. 3. Going along with the line of reasoning you've expressed, it seems that the user should call a 'lightbulb technician' when his Internet-enabled lightbulb is causing a problem. Do you really think that's realistic? 4. In most cases, the user won't have any idea which connected device is causing the problem. Expecting the user to determine this by trial-and-error is unrealistic; most people don't even understand how to troubleshoot electrical problems by trial-and-error, much less Internet-related problems. You are a self-selected specialist, and understand all these things and have a DIY attitude, because you're an expert in this field. Most people aren't experts in this field. Ask yourself how many people set up and use 2FA for any online service which supports it, on their own initiative (i.e., not having a bank ship them a pre provisioned dongle). The number of people capable of doing this troubleshooting for themselves is roughly equivalent to the number of people who've successfully set up 2FA on their own initiative. ----------------------------------- Roland Dobbins From rdobbins at arbor.net Tue Sep 27 05:35:40 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 12:35:40 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <9042FFC4-9E5D-4A35-8C11-BB908FCE41D1@llnw.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> <9042FFC4-9E5D-4A35-8C11-BB908FCE41D1@llnw.com> Message-ID: <7582CEBB-A9B7-4DC7-AA19-7AFCFAA396A6@arbor.net> On 27 Sep 2016, at 12:31, Jason Hofmann wrote: > It probably was a tough sell to get people to realize they were fully > responsible for their in-home wiring, but optional "inside wire > maintenance plans" made that clear while also adding to providers' > coffers. Perhaps something similar would work here. Concur that this is the least-improbable model, absolutely. But keep in mind that subscriptions/services for in-home wiring were (and are) also a tiny percentage of the user base. ----------------------------------- Roland Dobbins From lear at cisco.com Tue Sep 27 06:54:16 2016 From: lear at cisco.com (Eliot Lear) Date: Tue, 27 Sep 2016 08:54:16 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> Message-ID: <68205034-2027-e1ed-13ec-fb22eaa1a47e@cisco.com> John, On 9/27/16 2:13 AM, John R. Levine wrote: >> Therein lies the problem if the traffic does not look anomalous I >> suppose. But even if it does look unusual, ISPs would be asking >> consumers to trash/update/turn off a lot of devices in time ? like >> when every home has 10s or 100s of these devices. >> ISP: Dear customer, looks like one of your light switches is sending >> spam. >> Customer: Which one? I have 25 light switches. And 25 smart bulbs. >> And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and? > > That's why turning them off has to be mandatory if the ISP can't > mitigate the traffic in real time. As some on this thread know, I've been working with the folks who make light bulbs and switches. They fit a certain class of device that is not general purpose, but rather are specific in nature. For those devices it is possible for the manufacturers to inform the network what the communication pattern of the device is designed to be. This may be extraordinarily broad or quite narrow, depending on the device. Conveniently, the technology for describing much of this dates back to the paleolithic era in the form of access lists. That is what manufacturer usage descriptions are about. (Yep- MUD. There go my marketing credentials). They're slightly abstracted for adaptation to local deployments. There's a draft and we authors are soliciting comments. The service providers has a strong role to play here, since they drive standards for connectivity. Having some access to residential CPE in particular for this purpose, I believe, is very helpful because by doing so not only can SPs protect others, but can also provide lateral protection within the home. As I wrote upthread, if consumers come to see smart lightbulbs not just as useful, but also as a concern, they may wish their SPs to help them. That's the internalizing of an externality that I see possible, and maybe even probable over time. By the way, this isn't just about deliberate attacks. Ask Raul Rojas who built an IoT-based concept house and then had it taken down by a failing lightbulb.[2] Eliot [1] https://tools.ietf.org/html/draft-ietf-opsawg-mud-00 [2] http://fusion.net/story/55026/this-guys-light-bulb-ddosed-his-entire-smart-house/ > > Sorry, but something in your house is attacking strangers. Once you > figure out what it is, here's a handy list of links to the ongoing > class action suits against the manufacturers. > > Regards, > John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for > Dummies", > Please consider the environment before reading this e-mail. https://jl.ly > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From fw at deneb.enyo.de Tue Sep 27 08:51:24 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 10:51:24 +0200 Subject: Request for comment -- BCP38 In-Reply-To: <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> (Baldur Norddahl's message of "Mon, 26 Sep 2016 22:08:36 +0200") References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> Message-ID: <87y42dzpwz.fsf@mid.deneb.enyo.de> * Baldur Norddahl: > This means we can receive some packet on transit port A and then route out >>> a ICMP response on port B using the interface address from port A. But >>> transit B filters this ICMP packet because it has a source address >>> belonging to transit A. >> Interesting. But this looks like a feature request for the router >> vendor, and not like an issue with BCP 38 filtering as such. > > Can you quote an RFC for anything that the router is doing wrong? Is > there a requirement that a router must support source routing? It's not an RFC conformance issue (several implementations of source address selection are possible). But it appears to make it rather difficult to configure it in such a way it does what you need, and it looks like a reasonable enhancement request. > In our case we actually did contact the vendor. Turns out that it will > do source routing but not for packets from the control plane. There is > no way to resolve the issue with the current software available to > us. The vendor is not priotizing fixing this as I am also unable to > point to any RFC that is being violated. Source routing is not required to fix this. Other options are using a globally routed IP address for the source address (this can also be used to conserve address space because the interface addresses will not matter anymore), or chosing the interface address based on the outgoing interface. From fw at deneb.enyo.de Tue Sep 27 11:08:19 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 13:08:19 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160926234142.6E7705515473@rock.dv.isc.org> (Mark Andrews's message of "Tue, 27 Sep 2016 09:41:42 +1000") References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> Message-ID: <871t05zjks.fsf@mid.deneb.enyo.de> * Mark Andrews: > Dear customer, > we are seeing traffic coming from your network. > > If you need help isolating the source of the traffic here are a few > companies in your city that can help you. > > > > This is not a exhaustive list. > > Support We already had the problem in the past that customer notifications for compromised machines (with confirmed loss of user data, not just sourcing unexpected packets) looked like advertisements for antivirus products. To most customers, this looks like just another scam. From fw at deneb.enyo.de Tue Sep 27 11:19:54 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 13:19:54 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <68205034-2027-e1ed-13ec-fb22eaa1a47e@cisco.com> (Eliot Lear's message of "Tue, 27 Sep 2016 08:54:16 +0200") References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <68205034-2027-e1ed-13ec-fb22eaa1a47e@cisco.com> Message-ID: <87wphxy4h1.fsf@mid.deneb.enyo.de> * Eliot Lear: > As some on this thread know, I've been working with the folks who make > light bulbs and switches. They fit a certain class of device that is > not general purpose, but rather are specific in nature. For those > devices it is possible for the manufacturers to inform the network what > the communication pattern of the device is designed to be. This may be > extraordinarily broad or quite narrow, depending on the device. > Conveniently, the technology for describing much of this dates back to > the paleolithic era in the form of access lists. That is what > manufacturer usage descriptions are about. (Yep- MUD. There go my > marketing credentials). They're slightly abstracted for adaptation to > local deployments. There's a draft and we authors are soliciting comments. What's the end goal here? Charge extra if I'm connecting a TV instead of a light bulb? I'm not convinced that expected traffic profiles are the right answer. We already have that in the server hosting market, and it does constraint the types of services you can run on hosted servers (for the hosting providers who does this). I'm wary of the network putting severe constraints on application architecture, way beyond what is dictated by current technology. NAT more or less killed servers on consumer networks, and this kind of traffic profiling has begun to kill clients on server networks. > The service providers has a strong role to play here, since they drive > standards for connectivity. Having some access to residential CPE in > particular for this purpose, I believe, is very helpful because by doing > so not only can SPs protect others, but can also provide lateral > protection within the home. As I wrote upthread, if consumers come to > see smart lightbulbs not just as useful, but also as a concern, they may > wish their SPs to help them. That's the internalizing of an externality > that I see possible, and maybe even probable over time. Most service providers appear to be struggling to maintain up-to-date software on their CPEs (and their infrastructure), and following recommended configuration practices as they evolve. This does not give me confidence that they could perform device management at consumer scale. Do we know how much the average consumer trusts their ISP? Would they want their ISP to control the digital (and increasingly, physical) doors in their home? My own outlook is rather biased, unfortunately. From m4rtntns at gmail.com Tue Sep 27 11:20:04 2016 From: m4rtntns at gmail.com (Martin T) Date: Tue, 27 Sep 2016 14:20:04 +0300 Subject: nested prefixes in Internet Message-ID: Hi, let's assume that there is an ISP "A" operating in Europe region who has /19 IPv4 allocation from RIPE. From this /19 they have leased /24 to ISP "B" who is multi-homed. This means that ISP "B" would like to announce this /24 prefix to ISP "A" and also to ISP "C". AFAIK this gives two possibilities: 1) Deaggregate /19 in ISP "A" network and create "inetnum" and "route" objects for all those networks to RIPE database. This means that ISP "A" announces around dozen IPv4 prefixes to Internet except this /24 and ISP "B" announces this specific /24 to Internet. 2) ISP "A" continues to announce this /19 to Internet and at the same time ISP "B" starts to announce /24 to Internet. As this /24 is more-specific than /19, then traffic to hosts in this /24 will end up in ISP "B" network. Which approach is better? To me the second one seems to be better because it keeps the IPv4 routing-table smaller and requires ISP "A" to make no deaggregation related configuration changes. Only bit weird behavior I can see with the second option is that if ISP "B" stops for some reason announcing this /24 network to Internet, then traffic to hosts in this /24 gets to ISP "A" network and is blackholed there. thanks, Martin From jason.iannone at gmail.com Tue Sep 27 11:22:34 2016 From: jason.iannone at gmail.com (Jason Iannone) Date: Tue, 27 Sep 2016 07:22:34 -0400 Subject: Request for comment -- BCP38 In-Reply-To: <87y42dzpwz.fsf@mid.deneb.enyo.de> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> <87y42dzpwz.fsf@mid.deneb.enyo.de> Message-ID: I have a question regarding language. We've seen bcp38 described as a forwarding filter, preventing unallocated sources from leaving the AS. I understand that unicast reverse path forwarding checks support bcp38, but urpf is an input check with significant technical differences from output filters. Are urpf and bcp38 interchangeable terms in this discussion? It seems impractical and operationally risky to implement two unique ways to dos customers. What are the lessons learned by operators doing static output filters, strict urpf, or loose/feasible urpf? For a new implementation, I assume the safe bet is to start with loose urpf. Even if it stops only some traffic it at least gives the network to dip its toes and expose some customer brokenness. Bcp38 >From my allocation accept, else deny Urpf loose >From route table exist accept, else deny Urpf strict >From next hop interface true accept, else deny On Sep 27, 2016 4:52 AM, "Florian Weimer" wrote: > * Baldur Norddahl: > > > This means we can receive some packet on transit port A and then route > out > >>> a ICMP response on port B using the interface address from port A. But > >>> transit B filters this ICMP packet because it has a source address > >>> belonging to transit A. > >> Interesting. But this looks like a feature request for the router > >> vendor, and not like an issue with BCP 38 filtering as such. > > > > Can you quote an RFC for anything that the router is doing wrong? Is > > there a requirement that a router must support source routing? > > It's not an RFC conformance issue (several implementations of source > address selection are possible). But it appears to make it rather > difficult to configure it in such a way it does what you need, and it > looks like a reasonable enhancement request. > > > In our case we actually did contact the vendor. Turns out that it will > > do source routing but not for packets from the control plane. There is > > no way to resolve the issue with the current software available to > > us. The vendor is not priotizing fixing this as I am also unable to > > point to any RFC that is being violated. > > Source routing is not required to fix this. Other options are using a > globally routed IP address for the source address (this can also be > used to conserve address space because the interface addresses will > not matter anymore), or chosing the interface address based on the > outgoing interface. > From list at satchell.net Tue Sep 27 11:34:15 2016 From: list at satchell.net (Stephen Satchell) Date: Tue, 27 Sep 2016 04:34:15 -0700 Subject: Request for comment -- BCP38 In-Reply-To: <87y42dzpwz.fsf@mid.deneb.enyo.de> References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> <87y42dzpwz.fsf@mid.deneb.enyo.de> Message-ID: <7b2b0b6e-58ed-7f3a-f575-2f130942cdee@satchell.net> I'm trying to come up with a simple picture that embraces all the comments I've seen thus far on the definition of BCP38. The example scenario I'm about to paint may be over-simplified -- but I like to start simple. Given a single local inside network with: * multiple uplink providers (typical multi-home situation) * multiple edge routers, each connected to an upstream via a public routeable /30, and each further connected to the downstream inside network * 50 subnets (to pick a number) of routeable IP address space downstream from the edge routers, with routing announcements to the world that direct packets back to the edge routers BCP38 demands that ANY packet leaving ANY edge router to the upstream MUST have a source address: * within the 50 inside public route-able subnets, or * within a list of "my" addresses in the public /30 subnets. True statement? What am I missing here? (In this simplified view, I'm divorcing the BCP38 aspects from the practical effects of any policy or input filtering done by the upstreams, as I think that's a separate discussion -- important but off-topic right now for my understanding of BCP38 at its core. Those practical aspects can be added later, AFTER describing the basics.) From fw at deneb.enyo.de Tue Sep 27 11:36:39 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 13:36:39 +0200 Subject: Request for comment -- BCP38 In-Reply-To: (Jason Iannone's message of "Tue, 27 Sep 2016 07:22:34 -0400") References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> <87y42dzpwz.fsf@mid.deneb.enyo.de> Message-ID: <87shsly3p4.fsf@mid.deneb.enyo.de> * Jason Iannone: > I have a question regarding language. We've seen bcp38 described as a > forwarding filter, preventing unallocated sources from leaving the AS. I > understand that unicast reverse path forwarding checks support bcp38, but > urpf is an input check with significant technical differences from output > filters. > > Are urpf and bcp38 interchangeable terms in this discussion? It seems > impractical and operationally risky to implement two unique ways to dos > customers. What are the lessons learned by operators doing static output > filters, strict urpf, or loose/feasible urpf? Historically (in 1998, when RFC 2267 was released), BCP 38 was an egress filter applied at the AS boundary. A complainant would be able to identify the AS by looking at the IP address and contact the relevant ISP. Within the AS, that ISP could identify the actual packet source by other means. Reverse-path forwarding checks were explicitly ruled out as likely infeasible even in RFC 2267, except for links which are essentially dialups. Current terminology is more complicated. Personally, I think that mandating specific approaches such as BCP 38 or uRPF does not work. The goal should be to cut down spoofed traffic. Whether this is done by contracts (which are adhered to, obviously), monitoring or immediate technical enforcement in the routers, does not matter at all. > For a new implementation, I assume the safe bet is to start with loose > urpf. Even if it stops only some traffic it at least gives the network to > dip its toes and expose some customer brokenness. If loose uRPF is effective at all depends a lot on network architecture. From lear at cisco.com Tue Sep 27 11:52:06 2016 From: lear at cisco.com (Eliot Lear) Date: Tue, 27 Sep 2016 13:52:06 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <87wphxy4h1.fsf@mid.deneb.enyo.de> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <68205034-2027-e1ed-13ec-fb22eaa1a47e@cisco.com> <87wphxy4h1.fsf@mid.deneb.enyo.de> Message-ID: On 9/27/16 1:19 PM, Florian Weimer wrote: > * Eliot Lear: > >> As some on this thread know, I've been working with the folks who make >> light bulbs and switches. They fit a certain class of device that is >> not general purpose, but rather are specific in nature. For those >> devices it is possible for the manufacturers to inform the network what >> the communication pattern of the device is designed to be. This may be >> extraordinarily broad or quite narrow, depending on the device. >> Conveniently, the technology for describing much of this dates back to >> the paleolithic era in the form of access lists. That is what >> manufacturer usage descriptions are about. (Yep- MUD. There go my >> marketing credentials). They're slightly abstracted for adaptation to >> local deployments. There's a draft and we authors are soliciting comments. > What's the end goal here? Charge extra if I'm connecting a TV instead > of a light bulb? Not my end goal. My end goal is that consumers have a means to limit risk in their home environments, and service providers have a means to deliver that to them. > > I'm not convinced that expected traffic profiles are the right answer. > We already have that in the server hosting market, and it does > constraint the types of services you can run on hosted servers (for > the hosting providers who does this). I'm wary of the network putting > severe constraints on application architecture, way beyond what is > dictated by current technology. NAT more or less killed servers on > consumer networks, and this kind of traffic profiling has begun to > kill clients on server networks. The whole point of MUD is to leave control in the hands of those who have developed and have to support Things. It is not simply for the SP to decide what traffic is ok, or to charge more for it, but to respect the wishes of the developers. That may be sufficient to stop a lot of bad things from happening to a lot of Things. > >> The service providers has a strong role to play here, since they drive >> standards for connectivity. Having some access to residential CPE in >> particular for this purpose, I believe, is very helpful because by doing >> so not only can SPs protect others, but can also provide lateral >> protection within the home. As I wrote upthread, if consumers come to >> see smart lightbulbs not just as useful, but also as a concern, they may >> wish their SPs to help them. That's the internalizing of an externality >> that I see possible, and maybe even probable over time. > Most service providers appear to be struggling to maintain up-to-date > software on their CPEs (and their infrastructure), and following > recommended configuration practices as they evolve. This does not > give me confidence that they could perform device management at > consumer scale. It's NOT device management. Is network management. > > Do we know how much the average consumer trusts their ISP? Would they > want their ISP to control the digital (and increasingly, physical) > doors in their home? My own outlook is rather biased, unfortunately. > And again, this is the wrong way to look at it. The consumer should always get final say. They're the customer. This is a chance for the manufacturer of the device they're using to explain how the device is supposed to behave on the network. Example: how many times have you heard of some device leaving an extra service like SSH lying around? And would you really want that thermostat talking to ALL of the Internet or just locally + its cloud service? The manufacturer probably has a view about that, and I bet it aligns with the consumer and with the enterprise... Eliot -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From jared at puck.nether.net Tue Sep 27 12:20:22 2016 From: jared at puck.nether.net (Jared Mauch) Date: Tue, 27 Sep 2016 08:20:22 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: > On Sep 26, 2016, at 7:58 PM, Christopher Morrow wrote: > > On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews wrote: > >> >> Giving them real time access to the anomalous traffic log feed for >> their residence would also help. They or the specialist they bring >> in will be able to use that to trace back the problem. >> >> > wouldn't this work better as a standard bit of CPE software capability? > wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE > and kept for ~30mins (just guessing) in a circular buffer be 'good enough' > to present a pretty clear UI to the user? > > ip/mac/vendor sending (webtraffic|email|probes) to destination-name > [checkbox] > > > > select those youd' like to block [clickhere] > > This really doesn't seem hard, to present in a fairly straight forward > manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something > similar to this approach... but on the other hand: > "At least my ISP isn't snooping on all my traffic" The UBNT Edgerouter series has this. You can get fancy graphs and application breakdown. Scroll down and check the images: https://help.ubnt.com/hc/en-us/articles/204951104-EdgeMAX-Deep-Packet-Inspection-Engine-for-EdgeRouter You can see the hosts that are doing traffic and the destinations. They even have a model that takes a SFP so you can use it as CPE for FTTH. - Jared From list at satchell.net Tue Sep 27 12:31:24 2016 From: list at satchell.net (Stephen Satchell) Date: Tue, 27 Sep 2016 05:31:24 -0700 Subject: BCP38 adoption "incentives"? Message-ID: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".) From swmike at swm.pp.se Tue Sep 27 12:46:24 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Tue, 27 Sep 2016 14:46:24 +0200 (CEST) Subject: BCP38 adoption "incentives"? In-Reply-To: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: On Tue, 27 Sep 2016, Stephen Satchell wrote: > You have to make their ignorance SUBTRACT from the bottom line. I'd say there is no way to actually achieve this. BCP38 non-compliance doesn't hurt the one not in compliance in any significant amount, it hurts everybody else. The only way I can imagine BCP38 ever happening widely is by means of legislation, which of course is really hard because Internet spans countries/continents. Doing anti-spoofing should be done at the edge, the further up into the core you try to do it, the bigger risk you're breaking lots of users' connectivity, causing customer complaints. In some countries I'm sure BCP38 compliance could be increased by means of legislation and fining companies that do not do BCP38 filtering. But before we do that, we need to agree that BCP38 compliance is a must. I don't think we're there. I have heard people say that if they don't allow some of their customers to spoof, they're losing business, because some customers have built complete (deployed) solutions that are built on the fact that they can spoof packets. These people will have to be convinced that they're doing it wrong and re-design their solutions. This is going to cost them dearly, so they're going to be upset. With all the IoT devices out there, do people even need to spoof anymore? -- Mikael Abrahamsson email: swmike at swm.pp.se From zbynek at dialtelecom.cz Tue Sep 27 13:07:20 2016 From: zbynek at dialtelecom.cz (=?UTF-8?B?WmJ5bsSbayBQb3Nww61jaGFs?=) Date: Tue, 27 Sep 2016 15:07:20 +0200 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> The implementation of BCP38 over local market strongly increased after massive DDoS attacks in 2013 affecting major part of the industry thanks to an initiative of the most important local IXP. There is a special separate last-resort "island mode" network, which is intended to be activated in case of really major attacks to keep an accessibility of (at least) local services for local users. Implementation of BCP38 is one of the (lot of) requirements. Positive motivation is definitely better than asking politicians for regulations. More: https://fe.nix.cz/en/ Regards, Zbynek Dne 27.09.16 v 14:46 Mikael Abrahamsson napsal(a): > On Tue, 27 Sep 2016, Stephen Satchell wrote: > >> You have to make their ignorance SUBTRACT from the bottom line. > > I'd say there is no way to actually achieve this. BCP38 non-compliance > doesn't hurt the one not in compliance in any significant amount, it > hurts everybody else. > > The only way I can imagine BCP38 ever happening widely is by means of > legislation, which of course is really hard because Internet spans > countries/continents. > > Doing anti-spoofing should be done at the edge, the further up into the > core you try to do it, the bigger risk you're breaking lots of users' > connectivity, causing customer complaints. > re > In some countries I'm sure BCP38 compliance could be increased by means > of legislation and fining companies that do not do BCP38 filtering. But > before we do that, we need to agree that BCP38 compliance is a must. I > don't think we're there. I have heard people say that if they don't > allow some of their customers to spoof, they're losing business, because > some customers have built complete (deployed) solutions that are built > on the fact that they can spoof packets. These people will have to be > convinced that they're doing it wrong and re-design their solutions. > This is going to cost them dearly, so they're going to be upset. > > With all the IoT devices out there, do people even need to spoof anymore? > From fw at deneb.enyo.de Tue Sep 27 13:08:49 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 15:08:49 +0200 Subject: Request for comment -- BCP38 In-Reply-To: <7b2b0b6e-58ed-7f3a-f575-2f130942cdee@satchell.net> (Stephen Satchell's message of "Tue, 27 Sep 2016 04:34:15 -0700") References: <0c6509c7-6a67-0527-5fe4-3175a0553010@satchell.net> <78BDCC2C-0678-44F5-8367-0C89693F3F93@mykolab.com> <20160926144724.GD15891@sizone.org> <20160926151254.GH16464@bamboo.slabnet.com> <1894316783.1650.1474905622635.JavaMail.mhammett@ThunderFuck> <871t06sby2.fsf@mid.deneb.enyo.de> <52cdb2fe-ef51-a15a-d6a3-4cb55b0b3040@gmail.com> <87y42dzpwz.fsf@mid.deneb.enyo.de> <7b2b0b6e-58ed-7f3a-f575-2f130942cdee@satchell.net> Message-ID: <87bmz9v6am.fsf@mid.deneb.enyo.de> * Stephen Satchell: > Given a single local inside network with: > * multiple uplink providers (typical multi-home situation) > * multiple edge routers, each connected to an upstream via a public > routeable /30, and each further connected to the downstream inside > network > * 50 subnets (to pick a number) of routeable IP address space > downstream from the edge routers, with routing announcements to the > world that direct packets back to the edge routers > > BCP38 demands that ANY packet leaving ANY edge router to the upstream > MUST have a source address: > * within the 50 inside public route-able subnets, or > * within a list of "my" addresses in the public /30 subnets. > > True statement? This depends on the agreements with the upstream providers. They might reasonably exclude their own /30 they provided to you and the /30s from the other providers. In general, packets from the /30s would not travel far anyway because they would wail source address verification checks at the upstream provider. Some providers also use globally unique, but unrouted addresses for transfer networks, for infrastructure protection purposes. From list at satchell.net Tue Sep 27 13:14:29 2016 From: list at satchell.net (Stephen Satchell) Date: Tue, 27 Sep 2016 06:14:29 -0700 Subject: BCP38 -- disabusing misinformation in this discussion Message-ID: <8b46de9b-89a1-e36c-de8c-a18788b2b2f1@satchell.net> "BCP38 applies only to egress filtering" INCORRECT. The title of the update to BCP38/RFC2827, BCP84/RFC2074, exposes the balderdash on its face. That title? "Ingress Filtering for Multihomed Networks." Oops. This is a short snipping from the Introduction: > RFC 2827 recommends that ISPs police their customers' traffic by > dropping traffic entering their networks that is coming from a source > address not legitimately in use by the customer network. ENTERING, not LEAVING. QED (Although I can understand some of the confusion. Some people might IMPLEMENT BCP38 using egress filtering, relying on the infamous "as-if" rule. Indeed, in my implementation of BCP38, I implement it on both INPUT and OUTPUT, so that it traps nasty traffic generated within the four corners of the system on which the firewall runs. This works for servers, particularly servers with multiple virtual guests.) .......... Looking at RFC 2827 (I couldn't sleep, so here I am in the middle of the dark reading RFCs) Section 3, this paragraph appears: > In the example above, the attacker resides within 204.69.207.0/24, > which is provided Internet connectivity by ISP D. An input traffic > filter on the ingress (input) link of "router 2", which provides > connectivity to the attacker's network, restricts traffic to allow > only traffic originating from source addresses within the > 204.69.207.0/24 prefix, and prohibits an attacker from using > "invalid" source addresses which reside outside of this prefix range. > In other words, the ingress filter on "router 2" above would check: > > IF packet's source address from within 204.69.207.0/24 > THEN forward as appropriate > > IF packet's source address is anything else > THEN deny packet > > Network administrators should log information on packets which are > dropped. This then provides a basis for monitoring any suspicious > activity. (Note that "router 2" in the diagram is the router to which the attacker is talking, not a router on upstream ISPs A, B, C, and D. Although I guess router 2 could be owned by the ISP...) So, if I'm reading the RFC correctly, my original assertion that BCP mandates routers block "the bad stuff" on output, but ALSO on input. In fact, I don't see any discussion of output filtering in the RFC -- show me where it does, please. Now getting back to the discussion of uRPF (Unicast Reverse Path Forwarding), the RFC also talks about this: > ... The method > suggested is to look up source addresses to see that the return path > to that address would flow out the same interface as the packet > arrived upon. With the number of asymmetric routes in the Internet, > this would clearly be problematic. That description roughly corresponds with Cisco's definition of "strict". My question is whether my simplified example in a prior rock would correspond with Cisco's definition of "loose" -- you allow any source address in any downstream or local subnet the router knows about. ..... Just to show that I do read entire RFCs, here is this little nugget, from RFC 2827, Section 5: > If ingress filtering is used in an environment where DHCP or BOOTP is > used, the network administrator would be well advised to ensure that > packets with a source address of 0.0.0.0 and a destination of > 255.255.255.255 are allowed to reach the relay agent in routers when > appropriate. The scope of directed broadcast replication should be > controlled, however, and not arbitrarily forwarded. In the SOHO environment, the problem is moot because the DHCP is usually provided on the first router. (That's true in my home, and in businesses where I have installed networking.) In larger networks, or with ISPs that insist on doing DHCP upstream from a router, this is an issue. From swmike at swm.pp.se Tue Sep 27 13:17:37 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Tue, 27 Sep 2016 15:17:37 +0200 (CEST) Subject: BCP38 adoption "incentives"? In-Reply-To: <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> Message-ID: On Tue, 27 Sep 2016, Zbyn?k Posp?chal wrote: > The implementation of BCP38 over local market strongly increased after > massive DDoS attacks in 2013 affecting major part of the industry thanks > to an initiative of the most important local IXP. Hm, so the IX operator looks at packets at the IX (sFlow perhaps), see who is sending attack packets, and if they're spoofed, this ISP is then put in "quarantine", ie their IX port is basically now useless. That's an effective way of achieving local compliance. Wonder how this would work in other markets, commonly it's bad business to deny service to paying customers... But if most agree that this should be done, it's definitely a way to achieve compliance. -- Mikael Abrahamsson email: swmike at swm.pp.se From zbynek at dialtelecom.cz Tue Sep 27 13:29:01 2016 From: zbynek at dialtelecom.cz (=?UTF-8?B?WmJ5bsSbayBQb3Nww61jaGFs?=) Date: Tue, 27 Sep 2016 15:29:01 +0200 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> Message-ID: Dne 27.09.16 v 15:17 Mikael Abrahamsson napsal(a): > Hm, so the IX operator looks at packets at the IX (sFlow perhaps), see > who is sending attack packets, and if they're spoofed, this ISP is then > put in "quarantine", ie their IX port is basically now useless. Definitely not. Try to read first instead of speculations. Regards, Zbynek From fw at deneb.enyo.de Tue Sep 27 13:52:10 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 15:52:10 +0200 Subject: nested prefixes in Internet In-Reply-To: (Martin T.'s message of "Tue, 27 Sep 2016 14:20:04 +0300") References: Message-ID: <87shsl78mt.fsf@mid.deneb.enyo.de> * Martin T.: > let's assume that there is an ISP "A" operating in Europe region who > has /19 IPv4 allocation from RIPE. From this /19 they have leased /24 > to ISP "B" who is multi-homed. This means that ISP "B" would like to > announce this /24 prefix to ISP "A" and also to ISP "C". AFAIK this > gives two possibilities: > > 1) Deaggregate /19 in ISP "A" network and create "inetnum" and "route" > objects for all those networks to RIPE database. This means that ISP > "A" announces around dozen IPv4 prefixes to Internet except this /24 > and ISP "B" announces this specific /24 to Internet. > > 2) ISP "A" continues to announce this /19 to Internet and at the same > time ISP "B" starts to announce /24 to Internet. As this /24 is > more-specific than /19, then traffic to hosts in this /24 will end up > in ISP "B" network. > > > Which approach is better? Are the autonomous systems for the /19 and /24 connected directly? If they are, (2) is better from a global view (because it needs fewer routing table entries). (1) can be better from B's perspective because it prevents certain routing table optimizations (due to the lack of the covering prefix). But (1) can also be worse for B and A's other customers if /24s (and slightly shorter prefixes) in this part of the IPv4 address space are commonly filtered. From jsklein at gmail.com Tue Sep 27 13:52:38 2016 From: jsklein at gmail.com (Joe Klein) Date: Tue, 27 Sep 2016 09:52:38 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: What would it take to test for BCP38 for a specific AS? Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, Sep 27, 2016 at 8:31 AM, Stephen Satchell wrote: > Does anyone know if any upstream and tiered internet providers include in > their connection contracts a mandatory requirement that all > directly-connected routers be in compliance with BCP38? > > Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in > place internal policies requiring retail/business-customer-aggregating > routers to be in compliance with BCP38? > > Does any ISP, providing business Internet connectivity along with a block > of IP addresses, include language in their contracts that any directly > connected router must be in compliance with BCP38? > > I've seen a lot of moaning and groaning about how BCP38 is pretty much > being ignored. Education is one way to help, but that doesn't hit anyone > in the wallet. You have to motivate people to go out of their way to > *learn* about BCP38; most business people are too busy with things that > make them money to be concerned with "Internet esoterica" that doesn't add > to the bottom line. You have to make their ignorance SUBTRACT from the > bottom line. > > Contracts, properly enforced, can make a huge dent in the problem of BCP38 > adoption. At a number of levels. > > Equipment manufacturers not usually involved in this sort of thing (home > and SOHO market) would then have market incentive to provide equipment at > the low end that would provide BCP38 support. Especially equipment > manufacturers that incorporate embedded Linux in their products. They can > be creative in how they implement their product; let creativity blossom. > > I know, I know, BCP38 was originally directed at Internet Service > Providers at their edge to upstreams. I'm thinking that BCP38 needs to be > in place at any point -- every point? -- where you have a significant-sized > collection of systems/devices aggregated to single upstream connections. > Particular systems/devices where any source address can be generated and > propagated -- including compromised desktop computers, compromised light > bulbs, compromised wireless routers, compromised you-name-it. > > (That is one nice thing about NAT -- the bad guys can't build spoofed > packets. They *can* build, um, "other" packets...which is a different > subject entirely.) > > (N.B.: Now you know why I'm trying to get the simplest possible > definition of BCP38 into words. The RFCs don't contain "executive > summaries".) > From r.engehausen at gmail.com Tue Sep 27 14:05:18 2016 From: r.engehausen at gmail.com (Roy) Date: Tue, 27 Sep 2016 07:05:18 -0700 Subject: nested prefixes in Internet In-Reply-To: References: Message-ID: <72a17413-8ccf-1425-4953-277322e161af@gmail.com> Option 3? ISP A announces the /19 and the /24 while ISP B does just the /24 On 9/27/2016 4:20 AM, Martin T wrote: > Hi, > > let's assume that there is an ISP "A" operating in Europe region who > has /19 IPv4 allocation from RIPE. From this /19 they have leased /24 > to ISP "B" who is multi-homed. This means that ISP "B" would like to > announce this /24 prefix to ISP "A" and also to ISP "C". AFAIK this > gives two possibilities: > > 1) Deaggregate /19 in ISP "A" network and create "inetnum" and "route" > objects for all those networks to RIPE database. This means that ISP > "A" announces around dozen IPv4 prefixes to Internet except this /24 > and ISP "B" announces this specific /24 to Internet. > > 2) ISP "A" continues to announce this /19 to Internet and at the same > time ISP "B" starts to announce /24 to Internet. As this /24 is > more-specific than /19, then traffic to hosts in this /24 will end up > in ISP "B" network. > > > Which approach is better? To me the second one seems to be better > because it keeps the IPv4 routing-table smaller and requires ISP "A" > to make no deaggregation related configuration changes. Only bit weird > behavior I can see with the second option is that if ISP "B" stops for > some reason announcing this /24 network to Internet, then traffic to > hosts in this /24 gets to ISP "A" network and is blackholed there. > > > thanks, > Martin > From swmike at swm.pp.se Tue Sep 27 14:30:18 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Tue, 27 Sep 2016 16:30:18 +0200 (CEST) Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> Message-ID: On Tue, 27 Sep 2016, Zbyn?k Posp?chal wrote: > Dne 27.09.16 v 15:17 Mikael Abrahamsson napsal(a): > >> Hm, so the IX operator looks at packets at the IX (sFlow perhaps), see >> who is sending attack packets, and if they're spoofed, this ISP is then >> put in "quarantine", ie their IX port is basically now useless. > > Definitely not. Try to read first instead of speculations. The first page was completely devoid of any real technical information until I found the PDF (which from the color choice doesn't even look like a link). (https://www.nix.cz/cs/file/NIX_RULES_FENIX) It's still not obvious what the FENIX connection is used for from that PDF. It's called "last resort connection". What does that mean? Apart from that, it looks more like https://www.routingmanifesto.org/ in that organisations that have joined are stating that they will follow some operational guidelines (which make a lot of sense), but it's not that much more technical when it comes to inter-provider traffic. -- Mikael Abrahamsson email: swmike at swm.pp.se From swmike at swm.pp.se Tue Sep 27 14:32:42 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Tue, 27 Sep 2016 16:32:42 +0200 (CEST) Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: On Tue, 27 Sep 2016, Joe Klein wrote: > What would it take to test for BCP38 for a specific AS? Well, you can get people to run https://www.caida.org/projects/spoofer/#software I tried to get OpenWrt to include similar software, on by default, but some people are afraid that they might incur legal action on themselves by doing antispoofing-testing. https://www.ripe.net/participate/ripe/tf/anti-spoofing might be of interest. -- Mikael Abrahamsson email: swmike at swm.pp.se From bruns at 2mbit.com Tue Sep 27 14:48:24 2016 From: bruns at 2mbit.com (Brielle Bruns) Date: Tue, 27 Sep 2016 08:48:24 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> On 9/26/16 10:05 PM, Roland Dobbins wrote: > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. This will only work if all providers including cable, DSL and *shudders* WISPs (hate to be blunt, but who from my experience tend to be the least experienced and network knowledgeable people running a customer network) do it so customer's can't just switch networks and 'make the problem go away'. I use escalating price increases and delays in service/repair time on some of my consulting customers who do things I warned them to be more careful about. It takes time, but when $cost starts to become prohibitive, they stop and think. And the ones that never learn... Well, that's more $$$ in my pocket for the effort that I would normally charge otherwise. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From sam.silvester at gmail.com Tue Sep 27 05:17:16 2016 From: sam.silvester at gmail.com (Sam Silvester) Date: Tue, 27 Sep 2016 14:47:16 +0930 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: On Tue, Sep 27, 2016 at 1:35 PM, Roland Dobbins wrote: > It call comes down to the network operator, one way or another. There's > no separation in the public mind of 'my network' from 'the Internet' that > is analogous to the separation between 'the power company' and 'the > electrical wiring in my house/apartment' (and even in that space, the > conceptual separation often isn't present). > > Not sure I agree with this. To my knowledge, when somebody loses power, they go out and check circuit breakers and stuff, then either call an electrician (if a breaker doesn't stay on or the like), or call their electricity retailer/distributer. I'm not talking about IT / technically savvy people either. Now, I appreciate what you are saying though - end users are (generalisation incoming, and I am not having a go / being a dick toward end users) non-technical, busy and not willing to spend money on experts to help out. They don't understand that their ISP is not responsible / in control end to end etc, but yeah - not the best analogy above. As a second comment...I think there is something also to be considered in Mark's thoughts. NAT obviously breaks visibility from a network operator's perspective. As far as we can see, once a user is sending something flagged as abuse, the best we can tell is the public IPv4 address. This sucks, as it basically means suspend the user, who gets shitty as a result, and costs money and time on the phone to helpdesk as a result. In IPv6, it's not the case that all traffic is sourced from the same public IP, which is interesting, especially if the network operator's abuse desk has appropriate tooling to be able to marry that up to a device (probably with the end user involved of course, but maybe with less effort). I do also like the idea of IPv4 CPE having a menu displaying DHCP client ID, in/out bps/pps counters, especially if that is able to be exposed to the ISP helpdesk / abuse desk if needed. It's a nice to have, but not sure it'd ever get meaningful deployment in a timeframe that makes it useful. Food for thought. Sam From nanog at ics-il.net Tue Sep 27 15:30:32 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 27 Sep 2016 10:30:32 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160927044337.3586D5519DB9@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> Message-ID: <865859793.3192.1474990228118.JavaMail.mhammett@ThunderFuck> You must not support end users. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mark Andrews" To: "Roland Dobbins" Cc: nanog at nanog.org Sent: Monday, September 26, 2016 11:43:36 PM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey In message , Roland Dobbins wri tes: > > On 27 Sep 2016, at 6:58, Christopher Morrow wrote: > > > wouldn't something as simple as netflow/sflow/ipfix synthesized on the > > CPE and kept for ~30mins (just guessing) in a circular buffer be 'good > > enough' to present a pretty clear UI to the user? > > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different. > It call comes down to the network operator, one way or another. There's > no separation in the public mind of 'my network' from 'the Internet' > that is analogous to the separation between 'the power company' and 'the > electrical wiring in my house/apartment' (and even in that space, the > conceptual separation often isn't present). Actually I don't believe that. They do know what machines they have have connected to their home network. Boxes don't magically connect. Every machine was explictly connected. Mark > ----------------------------------- > Roland Dobbins -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From rdobbins at arbor.net Tue Sep 27 15:35:19 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 22:35:19 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: On 27 Sep 2016, at 21:48, Brielle Bruns wrote: > You start cutting off users or putting them into a walled garden until > they fix their machines, and they will start caring. It's important to keep in mind that in the not-so-distant future, their 'machines' will include every article of clothing they own, every can of soda in their refrigerator, ever major (and many minor) components of their automobiles, every blade in their windowshades, etc. ----------------------------------- Roland Dobbins From patrick at ianai.net Tue Sep 27 15:37:33 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Tue, 27 Sep 2016 11:37:33 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: On Sep 27, 2016, at 11:35 AM, Roland Dobbins wrote: > On 27 Sep 2016, at 21:48, Brielle Bruns wrote: >> You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. > > It's important to keep in mind that in the not-so-distant future, their 'machines' will include every article of clothing they own, every can of soda in their refrigerator, ever major (and many minor) components of their automobiles, every blade in their windowshades, etc. All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. If we try to teach them when a 6-pack of Coke is a DDoS source, we will have lost. -- TTFN, patrick From rdobbins at arbor.net Tue Sep 27 15:37:44 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 22:37:44 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: On 27 Sep 2016, at 12:17, Sam Silvester wrote: > or call their electricity retailer/distributer This is the problematic case that is, unfortunately, the default. People tend to view anything related to 'the Internet' as a utility, and for consumers and SMBs, they typically have a single provider, just as they typically do for electricity and water. ----------------------------------- Roland Dobbins From A.L.M.Buxey at lboro.ac.uk Tue Sep 27 15:38:46 2016 From: A.L.M.Buxey at lboro.ac.uk (Alan Buxey) Date: Tue, 27 Sep 2016 15:38:46 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <865859793.3192.1474990228118.JavaMail.mhammett@ThunderFuck> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org>, <865859793.3192.1474990228118.JavaMail.mhammett@ThunderFuck> Message-ID: hi, >From: NANOG on behalf of Mike Hammett >Sent: 27 September 2016 16:30 >Cc: nanog at nanog.org >Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey > >You must not support end users. haha...i read that wrong. I read it as a command, rather than an observation! ;-) alan From mel at beckman.org Tue Sep 27 15:40:19 2016 From: mel at beckman.org (Mel Beckman) Date: Tue, 27 Sep 2016 15:40:19 +0000 Subject: nested prefixes in Internet In-Reply-To: <72a17413-8ccf-1425-4953-277322e161af@gmail.com> References: , <72a17413-8ccf-1425-4953-277322e161af@gmail.com> Message-ID: Precisely. This is how it's done by providers I've worked with. -mel beckman > On Sep 27, 2016, at 7:06 AM, Roy wrote: > > > > Option 3? > > ISP A announces the /19 and the /24 while ISP B does just the /24 > >> On 9/27/2016 4:20 AM, Martin T wrote: >> Hi, >> >> let's assume that there is an ISP "A" operating in Europe region who >> has /19 IPv4 allocation from RIPE. From this /19 they have leased /24 >> to ISP "B" who is multi-homed. This means that ISP "B" would like to >> announce this /24 prefix to ISP "A" and also to ISP "C". AFAIK this >> gives two possibilities: >> >> 1) Deaggregate /19 in ISP "A" network and create "inetnum" and "route" >> objects for all those networks to RIPE database. This means that ISP >> "A" announces around dozen IPv4 prefixes to Internet except this /24 >> and ISP "B" announces this specific /24 to Internet. >> >> 2) ISP "A" continues to announce this /19 to Internet and at the same >> time ISP "B" starts to announce /24 to Internet. As this /24 is >> more-specific than /19, then traffic to hosts in this /24 will end up >> in ISP "B" network. >> >> >> Which approach is better? To me the second one seems to be better >> because it keeps the IPv4 routing-table smaller and requires ISP "A" >> to make no deaggregation related configuration changes. Only bit weird >> behavior I can see with the second option is that if ISP "B" stops for >> some reason announcing this /24 network to Internet, then traffic to >> hosts in this /24 gets to ISP "A" network and is blackholed there. >> >> >> thanks, >> Martin > From bruns at 2mbit.com Tue Sep 27 15:46:39 2016 From: bruns at 2mbit.com (Brielle Bruns) Date: Tue, 27 Sep 2016 09:46:39 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> On 9/27/16 9:35 AM, Roland Dobbins wrote: > On 27 Sep 2016, at 21:48, Brielle Bruns wrote: > >> You start cutting off users or putting them into a walled garden until >> they fix their machines, and they will start caring. > > It's important to keep in mind that in the not-so-distant future, their > 'machines' will include every article of clothing they own, every can of > soda in their refrigerator, ever major (and many minor) components of > their automobiles, every blade in their windowshades, etc. > I don't see how this is a problem exactly? If people want to buy devices that connect to their home network, they need to be aware of what these devices can do, and it is their responsibility. Better to teach them _now_ rather then later. If Timmy Numbnuts doesn't understand that plugging in a random device he found at Goodwill to his network could potentially carry liabilities, then he will keep doing it. I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From fw at deneb.enyo.de Tue Sep 27 15:49:52 2016 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 27 Sep 2016 17:49:52 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: (Roland Dobbins's message of "Tue, 27 Sep 2016 22:37:44 +0700") References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> Message-ID: <8737kl5om7.fsf@mid.deneb.enyo.de> * Roland Dobbins: > On 27 Sep 2016, at 12:17, Sam Silvester wrote: > >> or call their electricity retailer/distributer > > This is the problematic case that is, unfortunately, the default. > > People tend to view anything related to 'the Internet' as a utility, > and for consumers and SMBs, they typically have a single provider, > just as they typically do for electricity and water. Most people over here have at least two providers of water and Internet (although the second one is perhaps sufficient for brushing your teeth, but certainly not for a shower or a bath). From rdobbins at arbor.net Tue Sep 27 15:49:47 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 22:49:47 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote: > All the more reason to educate people TODAY on why having vulnerable > devices is a Very Bad Idea. Yes, but how do they determine that a given device is vulnerable? ----------------------------------- Roland Dobbins From rdobbins at arbor.net Tue Sep 27 16:05:32 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 23:05:32 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> Message-ID: <19495A87-B181-4F25-A757-E1721439A41C@arbor.net> On 27 Sep 2016, at 22:46, Brielle Bruns wrote: > I point to the current trend of parents watching and smiling, doing > nothing as their kids destroy people's stores and restaurants. ISPs > are literally doing the exact same thing when it comes to coddling > their customers. They can *see* the unruly children, but *choose* to ignore them. That's the difference. Keep in mind, most of the folks on this list are not representative of the average consumer in terms of the skill-sets which are relevant in this problem space. ----------------------------------- Roland Dobbins From patrick at ianai.net Tue Sep 27 16:05:58 2016 From: patrick at ianai.net (Patrick W. Gilmore) Date: Tue, 27 Sep 2016 12:05:58 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> Message-ID: <3DAC3247-AC1E-43E7-830E-E21E24C3849C@ianai.net> On Sep 27, 2016, at 11:49 AM, Roland Dobbins wrote: > On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote: >> All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. > > Yes, but how do they determine that a given device is vulnerable? Easy: Can you ping it? It?s vulnerable. :-) Hey, I said we would have to educate them. I did not say how that would happen. -- TTFN, patrick From rdobbins at arbor.net Tue Sep 27 16:07:19 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Tue, 27 Sep 2016 23:07:19 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <8737kl5om7.fsf@mid.deneb.enyo.de> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <8737kl5om7.fsf@mid.deneb.enyo.de> Message-ID: <2E2C59C6-FB0E-4A48-B9E5-F25A0B43A0E4@arbor.net> On 27 Sep 2016, at 22:49, Florian Weimer wrote: > Most people over here have at least two providers of water and > Internet (although the second one is perhaps sufficient for brushing > your teeth, but certainly not for a shower or a bath). That's not a common arrangement in much of the world, however. Especially the Internet part. ;> ----------------------------------- Roland Dobbins From keiths at neilltech.com Tue Sep 27 16:10:51 2016 From: keiths at neilltech.com (Keith Stokes) Date: Tue, 27 Sep 2016 16:10:51 +0000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> , <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> Message-ID: Assuming all devices are vulnerable isn't a bad start. -- Keith Stokes > On Sep 27, 2016, at 11:04 AM, Roland Dobbins wrote: > >> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote: >> >> All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. > > Yes, but how do they determine that a given device is vulnerable? > > ----------------------------------- > Roland Dobbins From beckman at angryox.com Tue Sep 27 16:13:39 2016 From: beckman at angryox.com (Peter Beckman) Date: Tue, 27 Sep 2016 12:13:39 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> Message-ID: On Tue, 27 Sep 2016, Brielle Bruns wrote: > I don't see how this is a problem exactly? If people want to buy devices > that connect to their home network, they need to be aware of what these > devices can do, and it is their responsibility. I understand that is what you want. What you might like. What we all would like. People taking responsibility for their impact on others. Unfortunately people plug things in, and if they work for them, they don't even think about how what they are doing might affect anyone else. In some cases, they don't even care. They've got soccer games and work and TV shows and kids and family. Who has time to become an expert in Internet security? Google is doing a great job of annoying or alerting customers to potential issues, such as the red lock icon on their email, indicating that the email was sent unencrypted. The user gets worried (oooh, a red lock, that must be bad, I'm going to yell at someone to fix it for me) and the service provider jumps to improve the Internet, ideally. FreeBSD updated their default config so you have to proactively remove email encryption. If we are truly worried about IoT and consumers contributing to the downfall of the Internet, force the consumer router manufacturers and third party firmware folks to implement whatever is necessary to make filters and blocking the default. 90%+ of consumers don't change any settings, beyond the SSID and Wifi Password, and those who do might take the responsibility you want. Get the ISPs to realize that secure-by-default consumer routers that they distribute saves them millions/billions of dollars annually in customer service and security personnel. Secure-by-default routers means cost-savings. Get ISPs to pressure manufacturers to implement measures to protect their own network and the Internet from the non-network-admin consumer. We tech folk need to do this for the Internet citizens who don't know, don't care, or don't have time to mess with it. > If Timmy Numbnuts doesn't understand that plugging in a random device he > found at Goodwill to his network could potentially carry liabilities, then he > will keep doing it. Timmy Numbnuts needs to be protected from himself, so when he plugs in that device, it doesn't do any harm to anyone but his own network. He'd have to proactively turn off features or filters on his Router in order to harm others. > I point to the current trend of parents watching and smiling, doing nothing > as their kids destroy people's stores and restaurants. ISPs are literally > doing the exact same thing when it comes to coddling their customers. Automation and default configs means customers don't have to do anything, nor think about it. They are protected both FROM harm from the Internet and FROM harming the Internet, at least by default. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- From dwcarder at es.net Tue Sep 27 15:43:54 2016 From: dwcarder at es.net (Dale W. Carder) Date: Tue, 27 Sep 2016 10:43:54 -0500 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> References: <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> Message-ID: <20160927154354.GC28235@cs-it-6805697.local> Thus spake Patrick W. Gilmore (patrick at ianai.net) on Sun, Sep 25, 2016 at 05:57:42PM -0400: > On Sep 25, 2016, at 5:50 PM, ryan landry wrote: > > On Sun, Sep 25, 2016 at 9:07 PM, Mark Andrews wrote: > > >> This is such a golden opportunity for each of you to find compromised > >> hosts on your network or your customer's network. The number of > >> genuine lookups of the blog vs the number of botted machine would > >> make it almost certain that anything directed at the blog is a > >> compromised machine. A phone call to the customer / further analysis > >> would reduce the false positive rate. > >> > >> Mark > >> > >> > > i wish you luck with that. explaining to grandma that her samsung smart tv > > has been rooted and needs to be updated should be good fun. > > > > for isp's it's a resourcing vs revenue problem. always has been. always > > will be. far more inclined to hold liable the folks that are churning out > > terribly dangerous cpe / IoT(shit). surely some regulatory body is looking > > into this. > > Yeah, ?cause that was so successful in the past. > > Remember University of Wisconsin vs. D-Link and their hard-coded NTP server address? Interestingly, this was just recently looked at again for the Internet of Things Software Update Workshop (IoTSU). See: http://pages.cs.wisc.edu/~plonka/iotsu/IoTSU_2016_paper_25.pdf 3,564 devices still remain. best, Dale From bruns at 2mbit.com Tue Sep 27 17:18:42 2016 From: bruns at 2mbit.com (Brielle Bruns) Date: Tue, 27 Sep 2016 11:18:42 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <19495A87-B181-4F25-A757-E1721439A41C@arbor.net> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> <19495A87-B181-4F25-A757-E1721439A41C@arbor.net> Message-ID: <59137182-f894-5824-2980-92808031b2ec@2mbit.com> On 9/27/16 10:05 AM, Roland Dobbins wrote: >> I point to the current trend of parents watching and smiling, doing >> nothing as their kids destroy people's stores and restaurants. ISPs >> are literally doing the exact same thing when it comes to coddling >> their customers. > > They can *see* the unruly children, but *choose* to ignore them. That's > the difference. I call shenanigans on providers not seeing their unruly users. They have no problems with bandwidth caps, or doing the dirty work of copyright police, both of which require some level of network monitoring per customer. > > Keep in mind, most of the folks on this list are not representative of > the average consumer in terms of the skill-sets which are relevant in > this problem space. I'm very well aware of that. I've got a fairly decent sized user base that I consult for, including small biz and end users, so I'm painfully aware of how... less then spectacular consumers are when it comes to even the most basic computer tasks. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From mike at mikejones.in Tue Sep 27 17:40:26 2016 From: mike at mikejones.in (Mike Jones) Date: Tue, 27 Sep 2016 18:40:26 +0100 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: On 27 September 2016 at 15:32, Mikael Abrahamsson wrote: > On Tue, 27 Sep 2016, Joe Klein wrote: > >> What would it take to test for BCP38 for a specific AS? > > > Well, you can get people to run > https://www.caida.org/projects/spoofer/#software > > I tried to get OpenWrt to include similar software, on by default, but some > people are afraid that they might incur legal action on themselves by doing > antispoofing-testing. Any network operator should know if their network is blocking it or not without having to deploy active probes across their network. If a network is thinking about it enough to want to block it, they will probably do so by turning knobs on their routers rather than deploying another patch to the CPE. I don't think the CPE is the solution here. - Mike Jones From bruns at 2mbit.com Tue Sep 27 17:50:02 2016 From: bruns at 2mbit.com (Brielle Bruns) Date: Tue, 27 Sep 2016 11:50:02 -0600 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <59137182-f894-5824-2980-92808031b2ec@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> <19495A87-B181-4F25-A757-E1721439A41C@arbor.net> <59137182-f894-5824-2980-92808031b2ec@2mbit.com> Message-ID: <1762f1a4-9700-f23e-a47a-6c929d0f4ad5@2mbit.com> On 9/27/16 11:18 AM, Brielle Bruns wrote: > On 9/27/16 10:05 AM, Roland Dobbins wrote: >>> I point to the current trend of parents watching and smiling, doing >>> nothing as their kids destroy people's stores and restaurants. ISPs >>> are literally doing the exact same thing when it comes to coddling >>> their customers. >> >> They can *see* the unruly children, but *choose* to ignore them. That's >> the difference. > > I call shenanigans on providers not seeing their unruly users. They > have no problems with bandwidth caps, or doing the dirty work of > copyright police, both of which require some level of network monitoring > per customer. Or even better example - the providers who monetize customer browsing/shopping habits using third party network device? Or SiteFinder like services with their DNS? Providers have no problems monitoring, intercepting, mangling. I know people here will say, "Well, I'm not like that/don't believe in that!", but we're not talking about technical decisions - but decisions made by people who's job is to squeeze as much profit out of the customers as possible. There is no financial incentive or penalty for preventing/limiting your customers from harming others. If there was, I don't think we'd be having this discussion. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From swmike at swm.pp.se Tue Sep 27 18:01:40 2016 From: swmike at swm.pp.se (Mikael Abrahamsson) Date: Tue, 27 Sep 2016 20:01:40 +0200 (CEST) Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: On Tue, 27 Sep 2016, Mike Jones wrote: > Any network operator should know if their network is blocking it or not > without having to deploy active probes across their network. Err... I was not referring to the operator doing this on the CPEs they provide to their customers. I was referring to enthusiast customers who are running their own CPEs to test if their ISPs are doing anti-spoofing properly or not, and report this information to someone centrally. -- Mikael Abrahamsson email: swmike at swm.pp.se From jsklein at gmail.com Tue Sep 27 18:16:23 2016 From: jsklein at gmail.com (Joe Klein) Date: Tue, 27 Sep 2016 14:16:23 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: The knobs that are available to push adoption of any standard can include "Doing nothing", "Educating the community", "Incentives", "Public Shaming", "Loss of business", "Engaging the policy & legal wanks". It seems to me the first two options have not moved the ball much. Must we move the last four to fix the DDOS problem? The last one scares me, but the other three might be valid method to move the ball. Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, Sep 27, 2016 at 10:32 AM, Mikael Abrahamsson wrote: > On Tue, 27 Sep 2016, Joe Klein wrote: > > What would it take to test for BCP38 for a specific AS? >> > > Well, you can get people to run https://www.caida.org/projects > /spoofer/#software > > I tried to get OpenWrt to include similar software, on by default, but > some people are afraid that they might incur legal action on themselves by > doing antispoofing-testing. > > https://www.ripe.net/participate/ripe/tf/anti-spoofing might be of > interest. > > > -- > Mikael Abrahamsson email: swmike at swm.pp.se > From rea+nanog at grid.kiae.ru Tue Sep 27 18:56:55 2016 From: rea+nanog at grid.kiae.ru (Eygene Ryabinkin) Date: Tue, 27 Sep 2016 21:56:55 +0300 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> References: <600470265.16530.1474778588492.JavaMail.zimbra@baylink.com> <235cdefe042147e4a14be955177c2e70@XCASPRD01-DFT.dpu.depaul.edu> <20160925120021.79280a95@p50.localdomain> <20160925210725.A27185504B03@rock.dv.isc.org> <3A96EAB7-BB50-495E-A75D-562E999D726D@ianai.net> Message-ID: <20160927185655.GP1375void.codelabs.ru@void.codelabs.ru> Sun, Sep 25, 2016 at 05:57:42PM -0400, Patrick W. Gilmore wrote: > Remember University of Wisconsin vs. D-Link and their hard-coded > NTP server address? UW vs Netgear and Poul-Henning Kamp vs D-Link, both on NTP stuff? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. From nanog at ics-il.net Tue Sep 27 19:49:30 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 27 Sep 2016 14:49:30 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: <785352154.109.1475005765275.JavaMail.mhammett@ThunderFuck> "who from my experience tend to be the least experienced and network knowledgeable people running a customer network" Also most likely to have built their network from scratch out of pure need (perhaps for themselves) rather than someone cashing in on a trend. No offense meant (though surely someone took it) either way. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Brielle Bruns" To: nanog at nanog.org Sent: Tuesday, September 27, 2016 9:48:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On 9/26/16 10:05 PM, Roland Dobbins wrote: > +1 for this capability in CPE. > > OTOH, it will be of no use whatsoever to the user. Providing the user > with access to anomalous traffic feeds won't help, either. > > Users aren't going to call in some third-party service/support company, > either. You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. This will only work if all providers including cable, DSL and *shudders* WISPs (hate to be blunt, but who from my experience tend to be the least experienced and network knowledgeable people running a customer network) do it so customer's can't just switch networks and 'make the problem go away'. I use escalating price increases and delays in service/repair time on some of my consulting customers who do things I warned them to be more careful about. It takes time, but when $cost starts to become prohibitive, they stop and think. And the ones that never learn... Well, that's more $$$ in my pocket for the effort that I would normally charge otherwise. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From jared at puck.nether.net Tue Sep 27 19:49:44 2016 From: jared at puck.nether.net (Jared Mauch) Date: Tue, 27 Sep 2016 15:49:44 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <20160927044337.3586D5519DB9@rock.dv.isc.org> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> Message-ID: > On Sep 27, 2016, at 12:43 AM, Mark Andrews wrote: > > Why not? You call a washing machine mechanic when the washing > machine plays up. This is not conceptually different. Mark, Your logic is infallible here, but the equivalencies are not. If I drive on the road and it?s bumpy, I would complain to the road people, but some people will take their car to the shop and says it shakes. When you are a toll-free call away from a complaint, often this barrier of proof is quite high. I recall something that Vijay said when he was still at AOL, if the customer phones in for support they lost all profit on the customer for the lifetime of the customer. Given that most people make decisions based on lowest cost (which isn?t always lowest or best due to marketing, promos, etc) the barrier for burden of proof is set such that a carrier must prove to a non-technical user it?s their fault. This proof is tough, not impossible, but look at your EDNS project, the problems are real and often can?t be easily addressed. - Jared From nanog at ics-il.net Tue Sep 27 19:51:40 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 27 Sep 2016 14:51:40 -0500 (CDT) Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> Message-ID: <947191292.114.1475005897064.JavaMail.mhammett@ThunderFuck> We can't teach other network operators the value of IPv6. Good luck teaching a consumer anything other than cat videos (and now recipes - unrelated to the former). ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Brielle Bruns" To: nanog at nanog.org Sent: Tuesday, September 27, 2016 10:46:39 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On 9/27/16 9:35 AM, Roland Dobbins wrote: > On 27 Sep 2016, at 21:48, Brielle Bruns wrote: > >> You start cutting off users or putting them into a walled garden until >> they fix their machines, and they will start caring. > > It's important to keep in mind that in the not-so-distant future, their > 'machines' will include every article of clothing they own, every can of > soda in their refrigerator, ever major (and many minor) components of > their automobiles, every blade in their windowshades, etc. > I don't see how this is a problem exactly? If people want to buy devices that connect to their home network, they need to be aware of what these devices can do, and it is their responsibility. Better to teach them _now_ rather then later. If Timmy Numbnuts doesn't understand that plugging in a random device he found at Goodwill to his network could potentially carry liabilities, then he will keep doing it. I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org From jared at puck.nether.net Tue Sep 27 19:59:20 2016 From: jared at puck.nether.net (Jared Mauch) Date: Tue, 27 Sep 2016 15:59:20 -0400 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> Message-ID: > On Sep 27, 2016, at 10:48 AM, Brielle Bruns wrote: > > You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. Wait until the user who claims perfection gets on the phone, etc. We had a network outage that caused a customer to go down, but they couldn?t sort it as they had two links to our network. Turns out they didn?t advertise the same routes on both links, and when I explained this to them they got very quiet on the phone. (It was a conference call with a lot of senior leadership on both sides involved). I asked if they wanted to correct the error so I could validate that it was fixed and they quickly went away. The tone before that point of the call was quite different. Residential customers can be far worse. I?ve heard of users clicking ?I cleaned my stuff? to get out of the portal nearly 200x because they are trained to ?Click to accept? everything else. This in no way shocks me, nor can everyone understand the complex error messages that computers provide. I myself end up searching for obscure error messages often due to hardware or software failures in $dayjob as well as supporting friends and family around me. End-users can be a unique problems to work with. - jared From nanog at ics-il.net Tue Sep 27 20:32:34 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 27 Sep 2016 15:32:34 -0500 (CDT) Subject: BCP38 adoption "incentives"? In-Reply-To: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" To: nanog at nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".) From Andrew.White2 at charter.com Tue Sep 27 20:44:35 2016 From: Andrew.White2 at charter.com (White, Andrew) Date: Tue, 27 Sep 2016 20:44:35 +0000 Subject: BCP38 adoption "incentives"? In-Reply-To: <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> Message-ID: Hi Mike, This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. Andrew NB: My personal opinion and not official communiqu? of Charter. Andrew White Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber andrew.white2 at charter.com Systems Engineer III, DAS DNS group Charter Communications 12405 Powerscourt Drive, St. Louis, MO 63131 -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Mike Hammett Sent: Tuesday, September 27, 2016 3:33 PM Cc: nanog at nanog.org Subject: Re: BCP38 adoption "incentives"? It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" To: nanog at nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".) From nanog at ics-il.net Tue Sep 27 20:51:30 2016 From: nanog at ics-il.net (Mike Hammett) Date: Tue, 27 Sep 2016 15:51:30 -0500 (CDT) Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> Message-ID: <2142792719.496.1475009488675.JavaMail.mhammett@ThunderFuck> They don't need to manage the router. The raw DSL modem, cable modem, etc. can watch the packets and see what's assigned. This would need new hardware, but it's not like this is happening quickly any other way. Yes, there are some consumer purchased DSL routers and cable routers, but doing what you can with what you can. FWIW, I believe most American ISPs *DO* manage their end-user routers. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Andrew White" To: "Mike Hammett" Cc: nanog at nanog.org Sent: Tuesday, September 27, 2016 3:44:35 PM Subject: RE: BCP38 adoption "incentives"? Hi Mike, This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. Andrew NB: My personal opinion and not official communiqu? of Charter. Andrew White Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber andrew.white2 at charter.com Systems Engineer III, DAS DNS group Charter Communications 12405 Powerscourt Drive, St. Louis, MO 63131 -----Original Message----- From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Mike Hammett Sent: Tuesday, September 27, 2016 3:33 PM Cc: nanog at nanog.org Subject: Re: BCP38 adoption "incentives"? It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" To: nanog at nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".) From beckman at angryox.com Tue Sep 27 21:20:48 2016 From: beckman at angryox.com (Peter Beckman) Date: Tue, 27 Sep 2016 17:20:48 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> Message-ID: On Tue, 27 Sep 2016, White, Andrew wrote: > This assumes the ISP manages the customer's CPE or home router, which is > often not the case. Adding such ACLs to the upstream device, operated by > the ISP, is not always easy or feasible. Which is why the manufacturer should deploy a default config which does this. Whatever the WAN IP, and by default, and in 90%+ configurations, there is a single WAN IP for CPE, ACLs are automatically managed to block all outbound packets that are NOT From: the WAN IP. And when DHCP or PPPoE gives a new IP, the rules are rewritten automatically by the CPE with updated rules. This won't fix the DDOS attach from IoT devices or IP Cameras or whatnot that don't attempt to hide their IP, but it would help with spoofing at the edge for the non-network saavy. > It would make sense for most ISPs to have egress filtering at the edge > (transit and peering points) to filter out packets that should not > originate from the ISP's ASN, although this does not prevent spoofing > between points in the ISP's network. Multi-tiered approaches are excellent. Start with the CPE, move to your aggs, then your big iron at the edges. Automate deployments and rule generation. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- From marka at isc.org Tue Sep 27 21:20:43 2016 From: marka at isc.org (Mark Andrews) Date: Wed, 28 Sep 2016 07:20:43 +1000 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: Your message of "Tue, 27 Sep 2016 15:49:44 -0400." References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <20160927044337.3586D5519DB9@rock.dv.isc.org> Message-ID: <20160927212043.E4B0F55203CA@rock.dv.isc.org> In message , Jared Mauch writes: > > > On Sep 27, 2016, at 12:43 AM, Mark Andrews wrote: > > > > Why not? You call a washing machine mechanic when the washing > > machine plays up. This is not conceptually different. > > Mark, > > Your logic is infallible here, but the equivalencies are not. If I > drive on the road and it???s bumpy, I would complain to the road people, > but some people will take their car to the shop and says it shakes. > > When you are a toll-free call away from a complaint, often this barrier > of proof is quite high. I recall something that Vijay said when he was > still at AOL, if the customer phones in for support they lost all profit > on the customer for the lifetime of the customer. > > Given that most people make decisions based on lowest cost (which isn???t > always lowest or best due to marketing, promos, etc) the barrier for > burden > of proof is set such that a carrier must prove to a non-technical user > it???s > their fault. > > This proof is tough, not impossible, but look at your EDNS project, the > problems are real and often can???t be easily addressed. Actually, EDNS shows they can be addressed. Firewall vendors are changing the defaults to allow through all packets that match the test classes. DNS vendors are fixing their products to properly handle packets with EDNS extension. DNS hosters are fixing their deployed firewalls and servers. Soon I'll be asking, my local opposition MP if she can ask why the DNS servers for *.gov.au aren't compliant with the standard after reporting to the DNS operators that they are broken. I suspect she will have fun with having more material to fling around. I'm having to reduce the parallelism of the test runs because the packets are being answered. Fixing EDNS is basically a education issue. Raise the awareness until it becomes something one can't ignore. Go look at the TLD graphs. Almost all the TLD operators have fixed their firewalls / servers. If Microsoft and Go Daddy fix their servers most of the incorrect echoing EDNS options and EDNS flags will disappear. Both have been informed. Microsoft about 2 years ago when we let them know that their servers have issues with EDNS, this included both the servers they ship in Windows and the servers answering DNS queries for Microsoft domains. They where reminded a year ago. Go Daddy was informed very recently (via email). Note that COOKIE is echoed. Also you can't report this to Microsoft using the email address listed below which was designed for reporting issues like this. Microsoft wants you to create a account or use twitter (which also requires an account to be created). You will note the DNS COOKIES are on by default. BIND 9.11.0 will be sending its queries with a DNS COOKIE option present. All your servers need to cope. % dig boimi.gov. @ns1-06.azure-dns.com soa ; <<>> DiG 9.11.0rc2 <<>> boimi.gov. @ns1-06.azure-dns.com soa ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54172 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ; COOKIE: ddcbdd73de5d5ef8 (echoed) ;; QUESTION SECTION: ;boimi.gov. IN SOA ;; ANSWER SECTION: boimi.gov. 3600 IN SOA ns1-06.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300 ;; ADDITIONAL SECTION: ns1-06.azure-dns.com. 3600 IN A 40.90.4.6 ;; Query time: 141 msec ;; SERVER: 40.90.4.6#53(40.90.4.6) ;; WHEN: Wed Sep 28 07:11:15 EST 2016 ;; MSG SIZE rcvd: 152 % % dig microsoft.com @ns1.msft.net ; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 7450 ;; flags: qr rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 5a294c21d4ac66a3 (echoed) ;; QUESTION SECTION: ;microsoft.com. IN A ;; Query time: 269 msec ;; SERVER: 2620:0:30::53#53(2620:0:30::53) ;; WHEN: Wed Sep 28 07:05:34 EST 2016 ;; MSG SIZE rcvd: 54 % dig microsoft.com @ns1.msft.net +nocookie ; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net +nocookie ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26221 ;; flags: qr aa rd; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;microsoft.com. IN A ;; ANSWER SECTION: microsoft.com. 3600 IN A 23.96.52.53 microsoft.com. 3600 IN A 191.239.213.197 microsoft.com. 3600 IN A 104.40.211.35 microsoft.com. 3600 IN A 104.43.195.251 microsoft.com. 3600 IN A 23.100.122.175 ;; Query time: 425 msec ;; SERVER: 2620:0:30::53#53(2620:0:30::53) ;; WHEN: Wed Sep 28 07:05:39 EST 2016 ;; MSG SIZE rcvd: 122 % Mark > - Jared -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From mh at xalto.net Tue Sep 27 17:52:54 2016 From: mh at xalto.net (Michael Hallgren) Date: Tue, 27 Sep 2016 19:52:54 +0200 Subject: nested prefixes in Internet In-Reply-To: References: , <72a17413-8ccf-1425-4953-277322e161af@gmail.com> Message-ID: Hi Martin, What do you want to do? Move from A to B or add A to B? Cheers, mh Le 27 sept. 2016 17:52, ? 17:52, Mel Beckman a ?crit: >Precisely. This is how it's done by providers I've worked with. > > -mel beckman > >> On Sep 27, 2016, at 7:06 AM, Roy wrote: >> >> >> >> Option 3? >> >> ISP A announces the /19 and the /24 while ISP B does just the /24 >> >>> On 9/27/2016 4:20 AM, Martin T wrote: >>> Hi, >>> >>> let's assume that there is an ISP "A" operating in Europe region who >>> has /19 IPv4 allocation from RIPE. From this /19 they have leased >/24 >>> to ISP "B" who is multi-homed. This means that ISP "B" would like to >>> announce this /24 prefix to ISP "A" and also to ISP "C". AFAIK this >>> gives two possibilities: >>> >>> 1) Deaggregate /19 in ISP "A" network and create "inetnum" and >"route" >>> objects for all those networks to RIPE database. This means that ISP >>> "A" announces around dozen IPv4 prefixes to Internet except this /24 >>> and ISP "B" announces this specific /24 to Internet. >>> >>> 2) ISP "A" continues to announce this /19 to Internet and at the >same >>> time ISP "B" starts to announce /24 to Internet. As this /24 is >>> more-specific than /19, then traffic to hosts in this /24 will end >up >>> in ISP "B" network. >>> >>> >>> Which approach is better? To me the second one seems to be better >>> because it keeps the IPv4 routing-table smaller and requires ISP "A" >>> to make no deaggregation related configuration changes. Only bit >weird >>> behavior I can see with the second option is that if ISP "B" stops >for >>> some reason announcing this /24 network to Internet, then traffic to >>> hosts in this /24 gets to ISP "A" network and is blackholed there. >>> >>> >>> thanks, >>> Martin >> From magicsata at gmail.com Wed Sep 28 00:18:25 2016 From: magicsata at gmail.com (Alistair Mackenzie) Date: Wed, 28 Sep 2016 01:18:25 +0100 Subject: Providing transit to unallocated networks Message-ID: Hi, I've come across a network which seem to be getting transit yet both the ASN and IP space is not allocated by the RIR. It does appear at some point that it was valid however this is no longer the case. The network is single homed and I tried asking the transit provider what their policy was on this but got no answer. Has anyone seen anything like this? What has happened in the past with things like this? Thanks, Alistair From bill at herrin.us Wed Sep 28 00:24:30 2016 From: bill at herrin.us (William Herrin) Date: Tue, 27 Sep 2016 20:24:30 -0400 Subject: Providing transit to unallocated networks In-Reply-To: References: Message-ID: On Tue, Sep 27, 2016 at 8:18 PM, Alistair Mackenzie wrote: > Hi, > > I've come across a network which seem to be getting transit yet both the > ASN and IP space is not allocated by the RIR. It does appear at some point > that it was valid however this is no longer the case. > > The network is single homed and I tried asking the transit provider what > their policy was on this but got no answer. > > Has anyone seen anything like this? What has happened in the past with > things like this? > > Thanks, > Alistair -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From bill at herrin.us Wed Sep 28 00:28:33 2016 From: bill at herrin.us (William Herrin) Date: Tue, 27 Sep 2016 20:28:33 -0400 Subject: Providing transit to unallocated networks In-Reply-To: References: Message-ID: On Tue, Sep 27, 2016 at 8:18 PM, Alistair Mackenzie wrote: > I've come across a network which seem to be getting transit yet both the > ASN and IP space is not allocated by the RIR. Hi Alistair, There is still unicast address space that isn't allocated by any RIR? Seriously though, check all your bases. Is not the space unallocated by all RIRs or just the one you expect to hold it? If you have a transit provider that's not playing by the rules, contact their transit providers to complain and if you still don't get satisfaction, I'd name and shame the lot of them. Failure to filter bad actors is how prefix hijacking happens. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From beecher at beecher.cc Wed Sep 28 00:37:40 2016 From: beecher at beecher.cc (Tom Beecher) Date: Tue, 27 Sep 2016 20:37:40 -0400 Subject: Providing transit to unallocated networks In-Reply-To: References: Message-ID: I've seen this with increasing frequency in the last 8-12 months, more with ASNs that were either expired/unallocated. Spammers seem to be snatching them up and hijacking IPs via bilateral peering to make it harder to notice. I've found it very difficult in some cases to get traction from IXes or transit providers to take action though. It makes sense ; they both have a financial interest in selling the ports and the pipes. In two cases this year, it took the threat of name and shame to get action taken, so it's a useful tool for sure. On Tue, Sep 27, 2016 at 8:18 PM, Alistair Mackenzie wrote: > Hi, > > I've come across a network which seem to be getting transit yet both the > ASN and IP space is not allocated by the RIR. It does appear at some point > that it was valid however this is no longer the case. > > The network is single homed and I tried asking the transit provider what > their policy was on this but got no answer. > > Has anyone seen anything like this? What has happened in the past with > things like this? > > Thanks, > Alistair > From magicsata at gmail.com Wed Sep 28 00:46:29 2016 From: magicsata at gmail.com (Alistair Mackenzie) Date: Wed, 28 Sep 2016 01:46:29 +0100 Subject: Providing transit to unallocated networks In-Reply-To: References: Message-ID: Thanks for this, it shows as apnic|ZZ|ipv4|103.***.***.0|1024|20160927|reserved||e-stats I expect this still stands with it being reserved? William, it's 100% an apnic range and shows no org and is registered to the APNIC Hostmaster. This applies for both the ASN and the address space. On 28 September 2016 at 01:28, William Herrin wrote: > On Tue, Sep 27, 2016 at 8:18 PM, Alistair Mackenzie wrote: > > I've come across a network which seem to be getting transit yet both the > > ASN and IP space is not allocated by the RIR. > > Hi Alistair, > > There is still unicast address space that isn't allocated by any RIR? > > Seriously though, check all your bases. Is not the space unallocated > by all RIRs or just the one you expect to hold it? If you have a > transit provider that's not playing by the rules, contact their > transit providers to complain and if you still don't get satisfaction, > I'd name and shame the lot of them. Failure to filter bad actors is > how prefix hijacking happens. > > Regards, > Bill Herrin > > > > -- > William Herrin ................ herrin at dirtside.com bill at herrin.us > Owner, Dirtside Systems ......... Web: > > On 28 September 2016 at 01:36, George Michaelson wrote: > check if the block is in this file. > > http://labs.apnic.net/delegated-nro-extended > > If not, then the block is hijacked or being abused. > > the file format is a bit obscure: the ipv4 record is base-ip|hostcount > but converting that to prefix length is pretty simple. > > -G > > On Wed, Sep 28, 2016 at 10:18 AM, Alistair Mackenzie > wrote: > > Hi, > > > > I've come across a network which seem to be getting transit yet both the > > ASN and IP space is not allocated by the RIR. It does appear at some > point > > that it was valid however this is no longer the case. > > > > The network is single homed and I tried asking the transit provider what > > their policy was on this but got no answer. > > > > Has anyone seen anything like this? What has happened in the past with > > things like this? > > > > Thanks, > > Alistair > From rdobbins at arbor.net Wed Sep 28 02:19:03 2016 From: rdobbins at arbor.net (Roland Dobbins) Date: Wed, 28 Sep 2016 09:19:03 +0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <59137182-f894-5824-2980-92808031b2ec@2mbit.com> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <3ae287cb-56b9-722c-fe47-720924818dfb@2mbit.com> <19495A87-B181-4F25-A757-E1721439A41C@arbor.net> <59137182-f894-5824-2980-92808031b2ec@2mbit.com> Message-ID: On 28 Sep 2016, at 0:18, Brielle Bruns wrote: > I call shenanigans on providers not seeing their unruly users. I was talking about the users, not the ISPs. ----------------------------------- Roland Dobbins From joelja at bogus.com Wed Sep 28 02:57:09 2016 From: joelja at bogus.com (joel jaeggli) Date: Tue, 27 Sep 2016 19:57:09 -0700 Subject: Providing transit to unallocated networks In-Reply-To: References: Message-ID: <386dcc99-7964-b195-4689-7d50305c8ec3@bogus.com> On 9/27/16 5:46 PM, Alistair Mackenzie wrote: > Thanks for this, it shows as > > apnic|ZZ|ipv4|103.***.***.0|1024|20160927|reserved||e-stats > > I expect this still stands with it being reserved? I'm not sure why you would bother obscuring it. What purpose does that serve in furthering the discussion? If it's not route-views>show ip bgp 103.6.232.0/22 BGP routing table entry for 103.6.232.0/22, version 87113221 Paths: (1 available, best #1, table default) Not advertised to any peer Refresh Epoch 1 3277 39710 20632 31133 58073 195.208.112.161 from 195.208.112.161 (194.85.4.4) Origin IGP, localpref 100, valid, external, best Community: 3277:39710 20632:65441 65535:65000 rx pathid: 0, tx pathid: 0x0 What is it? > > > William, it's 100% an apnic range and shows no org and is registered > to the APNIC Hostmaster. This applies for both the ASN and the address > space. > > > On 28 September 2016 at 01:28, William Herrin wrote: > >> On Tue, Sep 27, 2016 at 8:18 PM, Alistair Mackenzie wrote: >>> I've come across a network which seem to be getting transit yet both the >>> ASN and IP space is not allocated by the RIR. >> Hi Alistair, >> >> There is still unicast address space that isn't allocated by any RIR? >> >> Seriously though, check all your bases. Is not the space unallocated >> by all RIRs or just the one you expect to hold it? If you have a >> transit provider that's not playing by the rules, contact their >> transit providers to complain and if you still don't get satisfaction, >> I'd name and shame the lot of them. Failure to filter bad actors is >> how prefix hijacking happens. >> >> Regards, >> Bill Herrin >> >> >> >> -- >> William Herrin ................ herrin at dirtside.com bill at herrin.us >> Owner, Dirtside Systems ......... Web: >> >> > On 28 September 2016 at 01:36, George Michaelson wrote: > >> check if the block is in this file. >> >> http://labs.apnic.net/delegated-nro-extended >> >> If not, then the block is hijacked or being abused. >> >> the file format is a bit obscure: the ipv4 record is base-ip|hostcount >> but converting that to prefix length is pretty simple. >> >> -G >> >> On Wed, Sep 28, 2016 at 10:18 AM, Alistair Mackenzie >> wrote: >>> Hi, >>> >>> I've come across a network which seem to be getting transit yet both the >>> ASN and IP space is not allocated by the RIR. It does appear at some >> point >>> that it was valid however this is no longer the case. >>> >>> The network is single homed and I tried asking the transit provider what >>> their policy was on this but got no answer. >>> >>> Has anyone seen anything like this? What has happened in the past with >>> things like this? >>> >>> Thanks, >>> Alistair -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: OpenPGP digital signature URL: From outsider at scarynet.org Wed Sep 28 06:17:34 2016 From: outsider at scarynet.org (Alexander Maassen) Date: Wed, 28 Sep 2016 08:17:34 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey Message-ID: If those where in fact non-spoofed sources, then i am surely interested in getting that list in order to put it into my dnsbl (dronebl). So if someone has it, or can tell me who to contact. Feel free to provide me with it offlist. Especially if this botnet uses one of the many irc networks (like undernet) that is utilizing the dnsbl list and the cc is harbored there, it might help.? Also, most 'admins' only start realizing something is wrong in their network once their precious bizmail won't arrive at clients because their infected ip is listed and the remote mx refuses the mail because of the listing. Kind regards, Alexander Maassen - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer -------- Oorspronkelijk bericht --------Van: Hugo Slabbert Datum: 26-09-16 05:54 (GMT+01:00) Aan: "John R. Levine" Cc: nanog at nanog.org Onderwerp: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sun 2016-Sep-25 17:01:55 -0400, John R. Levine wrote: >>https://www.internetsociety.org/sites/default/files/01_5.pdf >> >>The attack is triggered by a few spoofs somewhere in the world. It is not >>feasible to stop this. > >That paper is about reflection attacks.? From what I've read, this was >not a reflection attack.? The IoT devices are infected with botware >which sends attack traffic directly.? Address spoofing is not particularly >useful for controlling botnets.? But that's not only remaining use of source address spoofing in direct attacks, no?? Even if reflection and amplification are not used, spoofing can still be used for obfuscation. >For example, the Conficker botnet generated pseudo-random domain names >where the bots looked for control traffic. > >>Please see https://www.ietf.org/rfc/rfc6561.txt > >Uh, yes, we're familiar with that.? We even know the people who wrote >it. It could use an update for IoT since I get the impression that in >many cases the only way for a nontechnical user to fix the infection >is to throw the device away. > >Regards, >John Levine, johnl at iecc.com, Primary Perpetrator of "The Internet for Dummies", >Please consider the environment before reading this e-mail. https://jl.ly -- Hugo Slabbert?????? | email, xmpp/jabber: hugo at slabnet.com pgp key: B178313E?? | also on Signal From lear at cisco.com Wed Sep 28 07:33:07 2016 From: lear at cisco.com (Eliot Lear) Date: Wed, 28 Sep 2016 09:33:07 +0200 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: <3DAC3247-AC1E-43E7-830E-E21E24C3849C@ianai.net> References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> <3DAC3247-AC1E-43E7-830E-E21E24C3849C@ianai.net> Message-ID: It's not just consumers that need to understand this. Manufacturers of Things are right now on a steep learning curve. Consider that thermostat, for just a moment. In The Gold Old Days, before it had a network interface, the manufacturer cared about a handful of things like at what temperature to turn the heat or A/C on maybe with some adjustments for time of day or day or week. And that was it. That is their domain of expertise. Not security. Now the Internet looks like a new shiny object that promises to provide some cool new world capabilities, like letting people adjust the temp while they're away, or using weather forecasts to manage hysteresis effects. And so, the manufacturer initially thinks, we'll add an interface to the product, and a little bit of code, and we're done. Now the manufacturer has stepped outside their domain of expertise, and doesn't have a full understanding of the risks that need to be addressed. We as experts in this domain can help by informing manufacturers of those risks. Eliot On 9/27/16 6:05 PM, Patrick W. Gilmore wrote: > On Sep 27, 2016, at 11:49 AM, Roland Dobbins wrote: >> On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote: >>> All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. >> Yes, but how do they determine that a given device is vulnerable? > Easy: Can you ping it? It?s vulnerable. > > :-) > > Hey, I said we would have to educate them. I did not say how that would happen. > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: OpenPGP digital signature URL: From ahebert at pubnix.net Wed Sep 28 11:05:57 2016 From: ahebert at pubnix.net (Alain Hebert) Date: Wed, 28 Sep 2016 07:05:57 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: <63b58694-db79-1f81-213a-410809ec4133@pubnix.net> Do not forget the "NRA" ways. Circular discussions every time an event arise, let it die out after a few days, and hopefully, nothing change. ----- Alain Hebert ahebert at pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 09/27/16 14:16, Joe Klein wrote: > The knobs that are available to push adoption of any standard can include > "Doing nothing", "Educating the community", "Incentives", "Public > Shaming", "Loss of business", "Engaging the policy & legal wanks". It seems > to me the first two options have not moved the ball much. > > Must we move the last four to fix the DDOS problem? The last one scares me, > but the other three might be valid method to move the ball. > > Joe Klein > "Inveniam viam aut faciam" > > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 > > On Tue, Sep 27, 2016 at 10:32 AM, Mikael Abrahamsson > wrote: > >> On Tue, 27 Sep 2016, Joe Klein wrote: >> >> What would it take to test for BCP38 for a specific AS? >> Well, you can get people to run https://www.caida.org/projects >> /spoofer/#software >> >> I tried to get OpenWrt to include similar software, on by default, but >> some people are afraid that they might incur legal action on themselves by >> doing antispoofing-testing. >> >> https://www.ripe.net/participate/ripe/tf/anti-spoofing might be of >> interest. >> >> >> -- >> Mikael Abrahamsson email: swmike at swm.pp.se >> From list at satchell.net Wed Sep 28 12:30:08 2016 From: list at satchell.net (Stephen Satchell) Date: Wed, 28 Sep 2016 05:30:08 -0700 Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey In-Reply-To: References: <20160926155649.14061.qmail@ary.lan> <20160926230946.685605514EDF@rock.dv.isc.org> <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com> <20160926234142.6E7705515473@rock.dv.isc.org> <20160926234939.B1961551553A@rock.dv.isc.org> <30d73fd7-e183-2a43-9929-67e039684023@2mbit.com> <9FCA63C8-3011-4391-B4C6-055EB0B75792@arbor.net> <3DAC3247-AC1E-43E7-830E-E21E24C3849C@ianai.net> Message-ID: <1f40a72d-037c-175d-7caf-15eb7aea1d20@satchell.net> On 09/28/2016 12:33 AM, Eliot Lear wrote: > It's not just consumers that need to understand this. Manufacturers of > Things are right now on a steep learning curve. Consider that > thermostat, for just a moment. In The Gold Old Days, before it had a > network interface, the manufacturer cared about a handful of things like > at what temperature to turn the heat or A/C on maybe with some > adjustments for time of day or day or week. And that was it. That is > their domain of expertise. Not security. > > Now the Internet looks like a new shiny object that promises to provide > some cool new world capabilities, like letting people adjust the temp > while they're away, or using weather forecasts to manage hysteresis > effects. And so, the manufacturer initially thinks, we'll add an > interface to the product, and a little bit of code, and we're done. Now > the manufacturer has stepped outside their domain of expertise, and > doesn't have a full understanding of the risks that need to be > addressed. We as experts in this domain can help by informing > manufacturers of those risks. Many manufacturers will outsource the Internet portion of their product to a software provider, not build from scratch "in house". The people we really need to get to are the ones that *provide* those packages the manufacturers use. In the case of embedded Linux solutions, the discussion need only be about what knobs to turn, and how far. From wesgeorge at puck.nether.net Wed Sep 28 15:08:00 2016 From: wesgeorge at puck.nether.net (Wesley George) Date: Wed, 28 Sep 2016 11:08:00 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: <2142792719.496.1475009488675.JavaMail.mhammett@ThunderFuck> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> <2142792719.496.1475009488675.JavaMail.mhammett@ThunderFuck> Message-ID: At least as far as cable is concerned, there is already configuration on the CMTS (e.g. https://www.cisco.com/c/en/us/support/docs/broadband-cable/cable-security/20691-source-verify.html ) that rejects things not coming from the assigned address, and AFAIK, it's best practice to enable it for more reasons than attack prevention. However... most residential IPv4 traffic lives behind a NATing CPE. The CPE will either: a) drop anything sourced from addresses not part of the configured LAN prefix b) NAT everything regardless of its source c) NAT things from its configured LAN, but bridge/forward anything else A and C result in spoofed traffic being dropped, either at the CPE or the CMTS. Same is true if the CPE itself has been compromised and is sending spoofed traffic. B results in it no longer being spoofed traffic, meaning that it defuses reflection attacks (the source address is no longer your attack target's address) but if it's raw packet floods, the attack still works but is now traceable back to its source. The behavior of a specific CPE is largely dependent on its raw source materials. Many CPE cheap plastic routers are built from a few common reference architectures from the chipset makers (Broadcom, Intel, etc) and then modified and adapted to brand their UI with the name silk-screened on the plastic, add features to distinguish one cheap plastic router from another, etc. Reasonably recent linux-based kernels do some of A by themselves, may even do things like RPF check, TCP sequence number window check, state comparison, so unless the CPE vendor defeats it when they adapt it for their use, it mostly works. Devices built to captive standards (i.e. purpose-built for Cable, DSL providers) could have specific guidance about which behavior is the correct one, but that may or may not affect what happens to the ones that show up at your favorite big box retailer. --Wes George, who has learned a thing or two about cable, but is speaking only for himself. > On Sep 27, 2016, at 4:51 PM, Mike Hammett wrote: > > They don't need to manage the router. The raw DSL modem, cable modem, etc. can watch the packets and see what's assigned. This would need new hardware, but it's not like this is happening quickly any other way. Yes, there are some consumer purchased DSL routers and cable routers, but doing what you can with what you can. > > FWIW, I believe most American ISPs *DO* manage their end-user routers. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > Midwest-IX > http://www.midwest-ix.com > > ----- Original Message ----- > > From: "Andrew White" > To: "Mike Hammett" > Cc: nanog at nanog.org > Sent: Tuesday, September 27, 2016 3:44:35 PM > Subject: RE: BCP38 adoption "incentives"? > > Hi Mike, > > This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. > > It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. > > Andrew > > NB: My personal opinion and not official communiqu? of Charter. > > > Andrew White > Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber > andrew.white2 at charter.com > Systems Engineer III, DAS DNS group > Charter Communications > 12405 Powerscourt Drive, St. Louis, MO 63131 > > > > -----Original Message----- > From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Mike Hammett > Sent: Tuesday, September 27, 2016 3:33 PM > Cc: nanog at nanog.org > Subject: Re: BCP38 adoption "incentives"? > > It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > Midwest-IX > http://www.midwest-ix.com > > ----- Original Message ----- > > From: "Stephen Satchell" > To: nanog at nanog.org > Sent: Tuesday, September 27, 2016 7:31:24 AM > Subject: BCP38 adoption "incentives"? > > Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? > > Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? > > Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? > > I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" > that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. > > Contracts, properly enforced, can make a huge dent in the problem of > BCP38 adoption. At a number of levels. > > Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. > > I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. > > (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) > > (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive > summaries".) > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From blake at ispn.net Wed Sep 28 15:59:51 2016 From: blake at ispn.net (Blake Hudson) Date: Wed, 28 Sep 2016 10:59:51 -0500 Subject: CDN Overload? In-Reply-To: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> References: <1067009549.11588.1474306487317.JavaMail.mhammett@ThunderFuck> Message-ID: <69186129-59a2-bf9d-1424-66bc1725878f@ispn.net> Mike, you might want to reference this thread - http://mailman.nanog.org/pipermail/nanog/2016-July/thread.html#87147 - as another data point. LLNW was sending data at levels ~ 10x greater than my policed DSL user's subscription rates. It seems to me that either the client or the server TCP stack was not working in a desirable manner. --Blake Mike Hammett wrote on 9/19/2016 12:34 PM: > I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. > > The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general issue and develop a better process for collecting what exactly is happening at the time and how to address it. > > One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended periods of time. > > An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the other side of the rate limiter. > > Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections downloading said data from Microsoft. > > The past week or two I've been hearing of people only having a single connection downloading at more than their plan rate. > > > These situations effectively shut out all other Internet traffic to that customer or even portion of the network for low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) > > > > > Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it to? > > > > > ----- > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > From mikevs at xs4all.net Wed Sep 28 18:39:42 2016 From: mikevs at xs4all.net (Miquel van Smoorenburg) Date: Wed, 28 Sep 2016 20:39:42 +0200 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> Message-ID: <201609281839.u8SIdgJh023114@xs9.xs4all.nl> In article you write: >What would it take to test for BCP38 for a specific AS? Well, if a certain browser vendor let the browser deduce the external IP address, then send out a UDP DNS PTR query for .in-addr.browser-vendor.com to say, a large DNS resolving cluster they also happen to be running, from a random source address, once every so often, they could very quickly build up a name-and-shame list of providers that don't do proper filtering. Just saying. Mike. From nanog at ics-il.net Wed Sep 28 19:05:55 2016 From: nanog at ics-il.net (Mike Hammett) Date: Wed, 28 Sep 2016 14:05:55 -0500 (CDT) Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> <2142792719.496.1475009488675.JavaMail.mhammett@ThunderFuck> Message-ID: <2028709844.551.1475089548255.JavaMail.mhammett@ThunderFuck> IPv6? Is that common in CMTSes or just in certain ones? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Wesley George" To: "Mike Hammett" Cc: nanog at nanog.org Sent: Wednesday, September 28, 2016 10:08:00 AM Subject: Re: BCP38 adoption "incentives"? At least as far as cable is concerned, there is already configuration on the CMTS (e.g. https://www.cisco.com/c/en/us/support/docs/broadband-cable/cable-security/20691-source-verify.html ) that rejects things not coming from the assigned address, and AFAIK, it's best practice to enable it for more reasons than attack prevention. However... most residential IPv4 traffic lives behind a NATing CPE. The CPE will either: a) drop anything sourced from addresses not part of the configured LAN prefix b) NAT everything regardless of its source c) NAT things from its configured LAN, but bridge/forward anything else A and C result in spoofed traffic being dropped, either at the CPE or the CMTS. Same is true if the CPE itself has been compromised and is sending spoofed traffic. B results in it no longer being spoofed traffic, meaning that it defuses reflection attacks (the source address is no longer your attack target's address) but if it's raw packet floods, the attack still works but is now traceable back to its source. The behavior of a specific CPE is largely dependent on its raw source materials. Many CPE cheap plastic routers are built from a few common reference architectures from the chipset makers (Broadcom, Intel, etc) and then modified and adapted to brand their UI with the name silk-screened on the plastic, add features to distinguish one cheap plastic router from another, etc. Reasonably recent linux-based kernels do some of A by themselves, may even do things like RPF check, TCP sequence number window check, state comparison, so unless the CPE vendor defeats it when they adapt it for their use, it mostly works. Devices built to captive standards (i.e. purpose-built for Cable, DSL providers) could have specific guidance about which behavior is the correct one, but that may or may not affect what happens to the ones that show up at your favorite big box retailer. --Wes George, who has learned a thing or two about cable, but is speaking only for himself. On Sep 27, 2016, at 4:51 PM, Mike Hammett < nanog at ics-il.net > wrote: They don't need to manage the router. The raw DSL modem, cable modem, etc. can watch the packets and see what's assigned. This would need new hardware, but it's not like this is happening quickly any other way. Yes, there are some consumer purchased DSL routers and cable routers, but doing what you can with what you can. FWIW, I believe most American ISPs *DO* manage their end-user routers. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Andrew White" < Andrew.White2 at charter.com > To: "Mike Hammett" < nanog at ics-il.net > Cc: nanog at nanog.org Sent: Tuesday, September 27, 2016 3:44:35 PM Subject: RE: BCP38 adoption "incentives"? Hi Mike, This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. It would make sense for most ISPs to have egress filtering at the edge (transit and peering points) to filter out packets that should not originate from the ISP's ASN, although this does not prevent spoofing between points in the ISP's network. Andrew NB: My personal opinion and not official communiqu? of Charter. Andrew White Desk: 314.394-9594 | Cell: 314-452-4386 | Jabber andrew.white2 at charter.com Systems Engineer III, DAS DNS group Charter Communications 12405 Powerscourt Drive, St. Louis, MO 63131 -----Original Message----- From: NANOG [ mailto:nanog-bounces at nanog.org ] On Behalf Of Mike Hammett Sent: Tuesday, September 27, 2016 3:33 PM Cc: nanog at nanog.org Subject: Re: BCP38 adoption "incentives"? It would be incredibly low impact to have the residential CPE block any source address not assigned by the ISP. Done. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Stephen Satchell" < list at satchell.net > To: nanog at nanog.org Sent: Tuesday, September 27, 2016 7:31:24 AM Subject: BCP38 adoption "incentives"? Does anyone know if any upstream and tiered internet providers include in their connection contracts a mandatory requirement that all directly-connected routers be in compliance with BCP38? Does anyone know if large ISPs like Comcast, Charter, or AT&T have put in place internal policies requiring retail/business-customer-aggregating routers to be in compliance with BCP38? Does any ISP, providing business Internet connectivity along with a block of IP addresses, include language in their contracts that any directly connected router must be in compliance with BCP38? I've seen a lot of moaning and groaning about how BCP38 is pretty much being ignored. Education is one way to help, but that doesn't hit anyone in the wallet. You have to motivate people to go out of their way to *learn* about BCP38; most business people are too busy with things that make them money to be concerned with "Internet esoterica" that doesn't add to the bottom line. You have to make their ignorance SUBTRACT from the bottom line. Contracts, properly enforced, can make a huge dent in the problem of BCP38 adoption. At a number of levels. Equipment manufacturers not usually involved in this sort of thing (home and SOHO market) would then have market incentive to provide equipment at the low end that would provide BCP38 support. Especially equipment manufacturers that incorporate embedded Linux in their products. They can be creative in how they implement their product; let creativity blossom. I know, I know, BCP38 was originally directed at Internet Service Providers at their edge to upstreams. I'm thinking that BCP38 needs to be in place at any point -- every point? -- where you have a significant-sized collection of systems/devices aggregated to single upstream connections. Particular systems/devices where any source address can be generated and propagated -- including compromised desktop computers, compromised light bulbs, compromised wireless routers, compromised you-name-it. (That is one nice thing about NAT -- the bad guys can't build spoofed packets. They *can* build, um, "other" packets...which is a different subject entirely.) (N.B.: Now you know why I'm trying to get the simplest possible definition of BCP38 into words. The RFCs don't contain "executive summaries".) From nick at foobar.org Wed Sep 28 19:24:41 2016 From: nick at foobar.org (Nick Hilliard) Date: Wed, 28 Sep 2016 20:24:41 +0100 Subject: BCP38 adoption "incentives"? In-Reply-To: <2028709844.551.1475089548255.JavaMail.mhammett@ThunderFuck> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> <2142792719.496.1475009488675.JavaMail.mhammett@ThunderFuck> <2028709844.551.1475089548255.JavaMail.mhammett@ThunderFuck> Message-ID: <57EC18F9.8030601@foobar.org> Mike Hammett wrote: > Is that common in CMTSes or just in certain ones? it's a mandatory part of the docsis3 specification. Nick From zbynek at dialtelecom.cz Wed Sep 28 20:26:38 2016 From: zbynek at dialtelecom.cz (=?UTF-8?B?WmJ5bsSbayBQb3Nww61jaGFs?=) Date: Wed, 28 Sep 2016 22:26:38 +0200 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <54c6f06c-0de2-c954-c404-58cd1a911cc5@dialtelecom.cz> Message-ID: Dne 27.09.16 v 16:30 Mikael Abrahamsson napsal(a): > The first page was completely devoid of any real technical information > until I found the PDF (which from the color choice doesn't even look > like a link). (https://www.nix.cz/cs/file/NIX_RULES_FENIX) > > It's still not obvious what the FENIX connection is used for from that > PDF. It's called "last resort connection". What does that mean? It means that a network suffering massive DDoS can switch off their transit and peering links and turn itself to anything like "island mode" operation, still keeping access to some of root and local TLD DNS servers, local content and/or local users. Fortunately, it has been never used except tests, but it is still an option when a network is in such kind of trouble. > Apart from that, it looks more like https://www.routingmanifesto.org/ in > that organisations that have joined are stating that they will follow > some operational guidelines (which make a lot of sense), but it's not > that much more technical when it comes to inter-provider traffic Routing manifesto cannot provide you anything in the oposite if you comply it, except, maybe, good feeling. Fenix (or Dutch Trusted Network Initiative) project can. Best Regards, Zbynek From Valdis.Kletnieks at vt.edu Wed Sep 28 20:40:37 2016 From: Valdis.Kletnieks at vt.edu (Valdis.Kletnieks at vt.edu) Date: Wed, 28 Sep 2016 16:40:37 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> Message-ID: <34748.1475095237@turing-police.cc.vt.edu> On Tue, 27 Sep 2016 20:44:35 -0000, "White, Andrew" said: > This assumes the ISP manages the customer's CPE or home router, which is > often not the case. Adding such ACLs to the upstream device, operated by the > ISP, is not always easy or feasible. Hopefully, if you've been burnt by this, you remembered to add it to the requirements list for the next time you buy customer-facing gear. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 484 bytes Desc: not available URL: From larrysheldon at cox.net Thu Sep 29 00:26:25 2016 From: larrysheldon at cox.net (larrysheldon at cox.net) Date: Wed, 28 Sep 2016 19:26:25 -0500 Subject: BCP38 adoption "incentives"? In-Reply-To: Message-ID: <20160928202625.7FHBY.225858.imail@fed1rmwml106> ---- Alain Hebert wrote: > Do not forget the "NRA" ways. I do not understand the "NRA" reference. From dwessels at verisign.com Thu Sep 29 15:15:23 2016 From: dwessels at verisign.com (Wessels, Duane) Date: Thu, 29 Sep 2016 15:15:23 +0000 Subject: Root Zone DNSSEC Operational Update -- ZSK length change In-Reply-To: References: Message-ID: <626768A0-5D22-4B97-8EFC-174894B7F297@verisign.com> A quick update on this change: A 2048-bit ZSK has been pre-published in the root zone as of September 20. We are not aware of any issues related to the appearance of the larger key. In less than 48 hours we will being publishing root zones signed with the 2048-bit ZSK. I will send another note once that has happened. If you observe any problems related to this change, please contact Verisign's customer service at info at verisign-grs.com. Duane W. > On Jul 28, 2016, at 3:37 PM, Wessels, Duane wrote: > > As you may know, Verisign, in its role as the Root Zone Maintainer > is also the operator of the root zone Zone Signing Key (ZSK). Later > this year, we will increase the size of the ZSK from 1024-bits to > 2048-bits. > > The root zone ZSK is normally rolled every calendar quarter, as per > our ?DNSSEC Practice Statement for the Root Zone ZSK operator.?[1] > The ZSK public keys are signed at quarterly key signing ceremonies > by ICANN in its role as the IANA Functions Operator. > > On September 20, 2016 the 2048-bit ZSK will be pre-published in the > root zone, following the standard ZSK rollover procedure. We intend > to begin publishing root zones signed with the first 2048-bit ZSK > on October 1, 2016. > > Some details of the ZSK size transition have recently been presented > at the DNS-OARC, NANOG, RIPE, ICANN, and IETF meetings.[2] If you > have any questions or concerns, please feel free to contact us at > zms at verisign.com. > > Please feel free to forward this message to anyone who might not have > seen it here. > > [1] https://www.verisign.com/assets/dps-zsk-operator-1532.pdf > [2] https://ripe72.ripe.net/wp-content/uploads/presentations/168-verisign-zsk-change.pdf > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bicknell at ufp.org Thu Sep 29 15:31:09 2016 From: bicknell at ufp.org (Leo Bicknell) Date: Thu, 29 Sep 2016 08:31:09 -0700 Subject: BCP38 adoption "incentives"? In-Reply-To: References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> Message-ID: <20160929153109.GA75993@ussenterprise.ufp.org> In a message written on Tue, Sep 27, 2016 at 08:44:35PM +0000, White, Andrew wrote: > This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. Unicast RFP should be a feature every ISP requires of all edge devices for at least 15 years now. It should be on by default for virtually all connections, and disabled only by request or when there are circumstances to suggest it would break things (e.g. a request for BGP with full tables over the link). At this point there's no excuse, anyone who has gear who can't do that has been asleep at the switch. It's been a standard feature in too much gear for too long. -- Leo Bicknell - bicknell at ufp.org PGP keys at http://www.ufp.org/~bicknell/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 811 bytes Desc: not available URL: From ahebert at pubnix.net Thu Sep 29 15:47:09 2016 From: ahebert at pubnix.net (Alain Hebert) Date: Thu, 29 Sep 2016 11:47:09 -0400 Subject: BCP38 adoption "incentives"? In-Reply-To: <20160929153109.GA75993@ussenterprise.ufp.org> References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> <20160929153109.GA75993@ussenterprise.ufp.org> Message-ID: Well there is money to be made in DDoS protection... See our "friends" still hosting "those" pay sites. Do not expect the vendors to cut themself of that market. ----- Alain Hebert ahebert at pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 09/29/16 11:31, Leo Bicknell wrote: > In a message written on Tue, Sep 27, 2016 at 08:44:35PM +0000, White, Andrew wrote: >> This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs to the upstream device, operated by the ISP, is not always easy or feasible. > Unicast RFP should be a feature every ISP requires of all edge > devices for at least 15 years now. It should be on by default for > virtually all connections, and disabled only by request or when > there are circumstances to suggest it would break things (e.g. a > request for BGP with full tables over the link). > > At this point there's no excuse, anyone who has gear who can't do > that has been asleep at the switch. It's been a standard feature > in too much gear for too long. > From drais at icantclick.org Wed Sep 28 23:25:13 2016 From: drais at icantclick.org (david raistrick) Date: Wed, 28 Sep 2016 19:25:13 -0400 Subject: contact @ detroit pistons IT/network org? Message-ID: by chance - anyone have a clueful contact at the detroit pistons who can help resolve an https MITM proxy problem? (likely a misconfigured watchguard.) trying to diagnose a proxy level certificate problem through a management level proxy is less than fun..... From math at sizone.org Thu Sep 29 19:46:57 2016 From: math at sizone.org (Ken Chase) Date: Thu, 29 Sep 2016 15:46:57 -0400 Subject: bogon identified? how to track down bogus IPs/ASN's Message-ID: <20160929194657.GE24544@sizone.org> My turn for the newb question: I've got a traceroute with this IP in it thats close to the end of the trace. 103.206.16.46 Chasing down this IP to see who the ISP a friend is using, figured out the diff between ARIN and APNIC whois for IPs (..bit of a learning curve, not sure why there's not just one whois interface syntax). whois -h whois.apnic.net -m 103.206.16.0/21 shows only the upper /22 being registered with APNIC (if you do -m on .16.0/22, there's no entry). So it seems to me these Ips arent registered properly with APNIC (could it be cross-registered with another RIR? Well it's not with ARIN who'd be the local.) But I do see this block in global bgp tables so it wasnt like someone decided to use 10.10.10/24 or 1.2.3/24 in their routing infrastructure. They're actually announcing; sh ip bg 103.206.16.0 ends in a path with 394786 135022 looking up 394786 I see avetria networks. looking up 135022 I see nothing at ARIN. At APNIC I get as-block: AS134557 - AS135580 descr: APNIC ASN block remarks: These AS numbers are further assigned by APNIC remarks: to APNIC members and end-users in the APNIC region but nothing more specific. However, this does show up in radb as avetria networks as well. (and various geolocate DBs put it in Melbourn.au though i know it's in use in Kitchener ontario). So what's not matching up here? /kc -- Ken Chase - math at sizone.org Guelph Ontario From arnold at nipper.de Thu Sep 29 19:48:35 2016 From: arnold at nipper.de (Arnold Nipper) Date: Thu, 29 Sep 2016 21:48:35 +0200 Subject: PeeringDB Product Committee Charter Survey / NANOG Message-ID: Hello PeeringDB users / hello NANOG, the PeeringDB Product Committee (PC, [0]) is charged with steering the future product development and running the market outreach efforts to continuously improve the value that PeeringDB delivers to the networks registered with PeeringDB, and the broader community. We're looking for feedback and input from the community on our charter proposal. Please take this short survey [1]. Your input and comments are appreciated! [0] http://docs.peeringdb.com/gov/ [1] https://www.surveymonkey.com/r/JN36DT2 Greetings Arnold -- Arnold Nipper email: arnold at nipper.de phone: +49 6224 5593407 2 mobile: +49 172 2650958 fax: +49 6224 5593407 9 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From fhr at fhrnet.eu Thu Sep 29 20:06:24 2016 From: fhr at fhrnet.eu (Filip Hruska) Date: Thu, 29 Sep 2016 22:06:24 +0200 Subject: bogon identified? how to track down bogus IPs/ASN's In-Reply-To: <20160929194657.GE24544@sizone.org> References: <20160929194657.GE24544@sizone.org> Message-ID: <05d6ad0e-90b9-edc5-7785-2d935c0ba441@fhrnet.eu> According to HE's BGP tool, the IP range is actually 103.206.16.0/22 and it looks like it's a bogon. http://bgp.he.net/net/103.206.16.0/22#_bogon Regards, Filip On 29.9.2016 21:46, Ken Chase wrote: > My turn for the newb question: > > I've got a traceroute with this IP in it thats close to the end of the trace. > > 103.206.16.46 > > Chasing down this IP to see who the ISP a friend is using, figured out > the diff between ARIN and APNIC whois for IPs (..bit of a learning curve, not > sure why there's not just one whois interface syntax). > > whois -h whois.apnic.net -m 103.206.16.0/21 > > shows only the upper /22 being registered with APNIC (if you do -m on > .16.0/22, there's no entry). > > So it seems to me these Ips arent registered properly with APNIC (could it > be cross-registered with another RIR? Well it's not with ARIN who'd be the local.) > > But I do see this block in global bgp tables so it wasnt like someone decided to use > 10.10.10/24 or 1.2.3/24 in their routing infrastructure. They're actually announcing; > > sh ip bg 103.206.16.0 ends in a path with 394786 135022 > > looking up 394786 I see avetria networks. looking up 135022 I see nothing at ARIN. > > At APNIC I get > > as-block: AS134557 - AS135580 > descr: APNIC ASN block > remarks: These AS numbers are further assigned by APNIC > remarks: to APNIC members and end-users in the APNIC region > > but nothing more specific. > > However, this does show up in radb as avetria networks as well. (and various geolocate > DBs put it in Melbourn.au though i know it's in use in Kitchener ontario). > > So what's not matching up here? > > /kc > -- > Ken Chase - math at sizone.org Guelph Ontario > From blake at ispn.net Thu Sep 29 20:13:21 2016 From: blake at ispn.net (Blake Hudson) Date: Thu, 29 Sep 2016 15:13:21 -0500 Subject: bogon identified? how to track down bogus IPs/ASN's In-Reply-To: <05d6ad0e-90b9-edc5-7785-2d935c0ba441@fhrnet.eu> References: <20160929194657.GE24544@sizone.org> <05d6ad0e-90b9-edc5-7785-2d935c0ba441@fhrnet.eu> Message-ID: <3e82aaf7-1959-092d-ff08-350c5d3a2402@ispn.net> As far as I can tell, AS394786 (Avetria Wireless) made up both AS135022 and the associated bogon IP ranges that AS announces (103.206.16.0/22 & 182.161.32.0/22) for its own use. Avetria's sole upstream provider appears to be AS54889 (Bluwest Inc). Probably an issue to discuss with both of these organizations. --Blake Filip Hruska wrote on 9/29/2016 3:06 PM: > According to HE's BGP tool, the IP range is actually 103.206.16.0/22 > and it looks like it's a bogon. > > http://bgp.he.net/net/103.206.16.0/22#_bogon > > Regards, > Filip > > On 29.9.2016 21:46, Ken Chase wrote: >> My turn for the newb question: >> >> I've got a traceroute with this IP in it thats close to the end of >> the trace. >> >> 103.206.16.46 >> >> Chasing down this IP to see who the ISP a friend is using, figured out >> the diff between ARIN and APNIC whois for IPs (..bit of a learning >> curve, not >> sure why there's not just one whois interface syntax). >> >> whois -h whois.apnic.net -m 103.206.16.0/21 >> >> shows only the upper /22 being registered with APNIC (if you do -m on >> .16.0/22, there's no entry). >> >> So it seems to me these Ips arent registered properly with APNIC >> (could it >> be cross-registered with another RIR? Well it's not with ARIN who'd >> be the local.) >> >> But I do see this block in global bgp tables so it wasnt like someone >> decided to use >> 10.10.10/24 or 1.2.3/24 in their routing infrastructure. They're >> actually announcing; >> >> sh ip bg 103.206.16.0 ends in a path with 394786 135022 >> >> looking up 394786 I see avetria networks. looking up 135022 I see >> nothing at ARIN. >> >> At APNIC I get >> >> as-block: AS134557 - AS135580 >> descr: APNIC ASN block >> remarks: These AS numbers are further assigned by APNIC >> remarks: to APNIC members and end-users in the APNIC region >> >> but nothing more specific. >> >> However, this does show up in radb as avetria networks as well. (and >> various geolocate >> DBs put it in Melbourn.au though i know it's in use in Kitchener >> ontario). >> >> So what's not matching up here? >> >> /kc >> -- >> Ken Chase - math at sizone.org Guelph Ontario >> From marka at isc.org Thu Sep 29 20:57:52 2016 From: marka at isc.org (Mark Andrews) Date: Fri, 30 Sep 2016 06:57:52 +1000 Subject: BCP38 adoption "incentives"? In-Reply-To: Your message of "Thu, 29 Sep 2016 11:47:09 -0400." References: <95921eaf-e215-9454-046e-375a86f19c33@satchell.net> <460080974.208.1475008352299.JavaMail.mhammett@ThunderFuck> <20160929153109.GA75993@ussenterprise.ufp.org> Message-ID: <20160929205752.3A4825546744@rock.dv.isc.org> Even if the customers are unaware of the spoofed traffic, ISPs should be aware which leaves them open for "aiding and abetting". This doesn't require inspecting the payload of the packets. This is the IP header which they are expected to examine and for which there is a BCP saying to drop spoofed packets. Sources are used for policy routing so the source field is expected to be processed. I would expect a Judge to take into consideration the BCP in deciding whether a ISP should be aware of the issue when deciding if a ISP is aiding and abetting by allowing spoofed packets to enter their network. Mark In message , Alain Hebert writes: > Well there is money to be made in DDoS protection... See our > "friends" still hosting "those" pay sites. > > Do not expect the vendors to cut themself of that market. > > ----- > Alain Hebert ahebert at pubnix.net > PubNIX Inc. > 50 boul. St-Charles > P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 > Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 > > On 09/29/16 11:31, Leo Bicknell wrote: > > In a message written on Tue, Sep 27, 2016 at 08:44:35PM +0000, White, Andrew wrote: > >> This assumes the ISP manages the customer's CPE or home router, which is often not the case. Adding such ACLs > to the upstream device, operated by the ISP, is not always easy or feasible. > > Unicast RFP should be a feature every ISP requires of all edge > > devices for at least 15 years now. It should be on by default for > > virtually all connections, and disabled only by request or when > > there are circumstances to suggest it would break things (e.g. a > > request for BGP with full tables over the link). > > > > At this point there's no excuse, anyone who has gear who can't do > > that has been asleep at the switch. It's been a standard feature > > in too much gear for too long. > > > -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka at isc.org From Mark_Feldman at comcast.com Thu Sep 29 18:59:07 2016 From: Mark_Feldman at comcast.com (Feldman, Mark) Date: Thu, 29 Sep 2016 18:59:07 +0000 Subject: Go Daddy contact Message-ID: Can someone from Go Daddy contact me off-list about a shared customer issue? Mark Comcast DNS From jcrowe215 at gmail.com Thu Sep 29 20:57:33 2016 From: jcrowe215 at gmail.com (J Crowe) Date: Thu, 29 Sep 2016 16:57:33 -0400 Subject: GoDaddy abuse Contact Message-ID: Could someone from GoDaddy please contact me off list? Running into issues where customers on our network are not able to reach their sites due to blackhole because of phishing sites that are on the shared IP. Thanks Joe Crowe Comcast From mike.lyon at gmail.com Fri Sep 30 02:43:19 2016 From: mike.lyon at gmail.com (mike.lyon at gmail.com) Date: Thu, 29 Sep 2016 19:43:19 -0700 Subject: TicketMaster / Live Nation admin on list? Message-ID: <90E7E839-135D-4997-93F3-9FE69DA14B4F@gmail.com> If so, can you contact me offlist? I seem to have a subnet that you guys don't like. Thank You, Mike From Bryan at bryanfields.net Fri Sep 30 16:49:34 2016 From: Bryan at bryanfields.net (Bryan Fields) Date: Fri, 30 Sep 2016 12:49:34 -0400 Subject: ARIN legacy block transfer process Message-ID: I'm trying to find a place on ARIN's website where this is addressed, but coming up short. I'm not the seller or buyer in this, but basically someone has a legacy block allocated by Postel and wants to sell the block as it's an owned asset. What's the process to get ARIN to move the admin/ownership of this? Do they only need to see a valid asset purchase agreement? There is no legacy RSA for this. I'm thinking of referring both parties to an experienced broker as well. Does anyone have current process experience with this? -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net From jared at puck.nether.net Fri Sep 30 17:06:38 2016 From: jared at puck.nether.net (Jared Mauch) Date: Fri, 30 Sep 2016 13:06:38 -0400 Subject: ARIN legacy block transfer process In-Reply-To: References: Message-ID: I would review this: https://ripe72.ripe.net/presentations/65-160524.ripe-transfer.pdf - Jared > On Sep 30, 2016, at 12:49 PM, Bryan Fields wrote: > > I'm trying to find a place on ARIN's website where this is addressed, but > coming up short. I'm not the seller or buyer in this, but basically someone > has a legacy block allocated by Postel and wants to sell the block as it's an > owned asset. > > What's the process to get ARIN to move the admin/ownership of this? Do they > only need to see a valid asset purchase agreement? There is no legacy RSA for > this. > > I'm thinking of referring both parties to an experienced broker as well. > > Does anyone have current process experience with this? > -- > Bryan Fields > > 727-409-1194 - Voice > http://bryanfields.net From bill at herrin.us Fri Sep 30 17:22:19 2016 From: bill at herrin.us (William Herrin) Date: Fri, 30 Sep 2016 13:22:19 -0400 Subject: ARIN legacy block transfer process In-Reply-To: References: Message-ID: On Fri, Sep 30, 2016 at 12:49 PM, Bryan Fields wrote: > I'm trying to find a place on ARIN's website where this is addressed, but > coming up short. I'm not the seller or buyer in this, but basically someone > has a legacy block allocated by Postel and wants to sell the block as it's an > owned asset. > > What's the process to get ARIN to move the admin/ownership of this? Hi Bryan, If this is entirely within the ARIN region, the process is this: 1. Buyer gets approved by ARIN for a specified transfer in the amount to be purchased. Signs RSA. 2. Legacy registrant (seller) signs the LRSA, proves identity and authority to ARIN, places legacy block under the LRSA. 3. Buyer requests specified transfer from ARIN, seller authorizes it. 4. ARIN transfers registration to buyer, now under the RSA. Note that you can't sell the block as an "owned asset" and have ARIN recognize the change. ARIN does not recognize ownership of IP address blocks, they only recognize registration and authorized agents. Note that if the legacy block's registration is defective, you're in for some trouble. Defects include a defunct organization where an individual who is not the registrant has maintained the block. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From sethm at rollernet.us Fri Sep 30 17:26:36 2016 From: sethm at rollernet.us (Seth Mattinen) Date: Fri, 30 Sep 2016 10:26:36 -0700 Subject: ARIN legacy block transfer process In-Reply-To: References: Message-ID: <20c65b90-59d6-6e48-caae-8aa58985f0e7@rollernet.us> On 9/30/16 09:49, Bryan Fields wrote: > I'm trying to find a place on ARIN's website where this is addressed, but > coming up short. I'm not the seller or buyer in this, but basically someone > has a legacy block allocated by Postel and wants to sell the block as it's an > owned asset. > > What's the process to get ARIN to move the admin/ownership of this? Do they > only need to see a valid asset purchase agreement? There is no legacy RSA for > this. It'll have to go under RSA to stay with ARIN. Or you can do a transfer to RIPE. ~Seth From Bryan at bryanfields.net Fri Sep 30 17:34:44 2016 From: Bryan at bryanfields.net (Bryan Fields) Date: Fri, 30 Sep 2016 13:34:44 -0400 Subject: ARIN legacy block transfer process In-Reply-To: References: Message-ID: <8c41e580-9d97-c919-41bb-c88ff2684a84@bryanfields.net> On 9/30/16 1:22 PM, William Herrin wrote: > Note that you can't sell the block as an "owned asset" and have ARIN > recognize the change. ARIN does not recognize ownership of IP address > blocks, they only recognize registration and authorized agents. This would seem to be in violation of what the NSF has said about this space. I thought ARIN was slapped hard once before about this very thing? Thanks, -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net From jim at reptiles.org Fri Sep 30 17:42:03 2016 From: jim at reptiles.org (Jim Mercer) Date: Fri, 30 Sep 2016 13:42:03 -0400 Subject: ARIN legacy block transfer process In-Reply-To: References: Message-ID: <20160930174203.GA22877@reptiles.org> On Fri, Sep 30, 2016 at 01:22:19PM -0400, William Herrin wrote: > 1. Buyer gets approved by ARIN for a specified transfer in the amount > to be purchased. Signs RSA. > 2. Legacy registrant (seller) signs the LRSA, proves identity and > authority to ARIN, places legacy block under the LRSA. i don't recall having to sign an LRSA, but they certainly do need proof that you are the entity that holds the block. > 3. Buyer requests specified transfer from ARIN, seller authorizes it. > 4. ARIN transfers registration to buyer, now under the RSA. if you are a legacy holder, and you want a dead simple way to sell off your rights to blocks, register for the STLS, and get your blocks added to the list. https://www.arin.net/public/secure/downloads/index.xhtml (at the bottom) this is also a good place to go if you want to buy rights to a block. you can also solicit offers to buy from anyone on the list. works well, and less dodgy-ness, since everyone on the list has been vetted (to some degree) by ARIN. --jim -- Jim Mercer Reptilian Research jim at reptiles.org +1 416 410-5633 Life should not be a journey to the grave with the intention of arriving safely in a pretty and well preserved body, but rather to skid in broadside in a cloud of smoke, thoroughly used up, totally worn out, and loudly proclaiming "Wow! What a Ride!" -- Hunter S. Thompson From erey at ernw.de Fri Sep 30 17:47:45 2016 From: erey at ernw.de (Enno Rey) Date: Fri, 30 Sep 2016 19:47:45 +0200 Subject: ARIN legacy block transfer process In-Reply-To: <20c65b90-59d6-6e48-caae-8aa58985f0e7@rollernet.us> References: <20c65b90-59d6-6e48-caae-8aa58985f0e7@rollernet.us> Message-ID: <20160930174745.GF12280@ernw.de> Hi, On Fri, Sep 30, 2016 at 10:26:36AM -0700, Seth Mattinen wrote: > On 9/30/16 09:49, Bryan Fields wrote: > > I'm trying to find a place on ARIN's website where this is addressed, but > > coming up short. I'm not the seller or buyer in this, but basically someone > > has a legacy block allocated by Postel and wants to sell the block as it's an > > owned asset. > > > > What's the process to get ARIN to move the admin/ownership of this? Do they > > only need to see a valid asset purchase agreement? There is no legacy RSA for > > this. > > > It'll have to go under RSA to stay with ARIN. > > Or you can do a transfer to RIPE. carefully note the "or". If you first move it under ARIN's jurisdiction (by signing an RSA) and *then* transfer it to RIPE, it won't be "legacy" any more in the course of the 2nd step and RIPE's 2-yr holding period comes into play (=> it can't be transferred during that time). Note also there's voices recommending not to sign an RSA for legacy space (in certain situations, at least), see http://ipv4marketgroup.com/dont-sign-an-rsa-during-your-82-ipv4-transfer/. best Enno > > ~Seth -- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator ======================================================= From cscora at apnic.net Fri Sep 30 18:01:31 2016 From: cscora at apnic.net (Routing Analysis Role Account) Date: Sat, 1 Oct 2016 04:01:31 +1000 (AEST) Subject: Weekly Routing Table Report Message-ID: <20160930180132.09C60C453D@thyme.apnic.net> This is an automated weekly mailing describing the state of the Internet Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, SAFNOG, SdNOG, BJNOG, CaribNOG and the RIPE Routing WG. Daily listings are sent to bgp-stats at lists.apnic.net For historical data, please see http://thyme.rand.apnic.net. If you have any comments please contact Philip Smith . Routing Table Report 04:00 +10GMT Sat 01 Oct, 2016 Report Website: http://thyme.rand.apnic.net Detailed Analysis: http://thyme.rand.apnic.net/current/ Analysis Summary ---------------- BGP routing table entries examined: 612615 Prefixes after maximum aggregation (per Origin AS): 220315 Deaggregation factor: 2.78 Unique aggregates announced (without unneeded subnets): 298785 Total ASes present in the Internet Routing Table: 54938 Prefixes per ASN: 11.15 Origin-only ASes present in the Internet Routing Table: 36401 Origin ASes announcing only one prefix: 15384 Transit ASes present in the Internet Routing Table: 6509 Transit-only ASes present in the Internet Routing Table: 168 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 54 Max AS path prepend of ASN ( 55644) 51 Prefixes from unregistered ASNs in the Routing Table: 62 Unregistered ASNs in the Routing Table: 16 Number of 32-bit ASNs allocated by the RIRs: 15608 Number of 32-bit ASNs visible in the Routing Table: 12028 Prefixes from 32-bit ASNs in the Routing Table: 48261 Number of bogon 32-bit ASNs visible in the Routing Table: 151 Special use prefixes present in the Routing Table: 0 Prefixes being announced from unallocated address space: 333 Number of addresses announced to Internet: 2829857764 Equivalent to 168 /8s, 172 /16s and 51 /24s Percentage of available address space announced: 76.4 Percentage of allocated address space announced: 76.4 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 98.3 Total number of prefixes smaller than registry allocations: 199109 APNIC Region Analysis Summary ----------------------------- Prefixes being announced by APNIC Region ASes: 156521 Total APNIC prefixes after maximum aggregation: 42983 APNIC Deaggregation factor: 3.64 Prefixes being announced from the APNIC address blocks: 170242 Unique aggregates announced from the APNIC address blocks: 69342 APNIC Region origin ASes present in the Internet Routing Table: 5193 APNIC Prefixes per ASN: 32.78 APNIC Region origin ASes announcing only one prefix: 1146 APNIC Region transit ASes present in the Internet Routing Table: 943 Average APNIC Region AS path length visible: 4.4 Max APNIC Region AS path length visible: 54 Number of APNIC region 32-bit ASNs visible in the Routing Table: 2395 Number of APNIC addresses announced to Internet: 760195012 Equivalent to 45 /8s, 79 /16s and 167 /24s APNIC AS Blocks 4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-137529 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary ---------------------------- Prefixes being announced by ARIN Region ASes: 184711 Total ARIN prefixes after maximum aggregation: 89553 ARIN Deaggregation factor: 2.06 Prefixes being announced from the ARIN address blocks: 190560 Unique aggregates announced from the ARIN address blocks: 88497 ARIN Region origin ASes present in the Internet Routing Table: 16181 ARIN Prefixes per ASN: 11.78 ARIN Region origin ASes announcing only one prefix: 5681 ARIN Region transit ASes present in the Internet Routing Table: 1703 Average ARIN Region AS path length visible: 3.8 Max ARIN Region AS path length visible: 23 Number of ARIN region 32-bit ASNs visible in the Routing Table: 1490 Number of ARIN addresses announced to Internet: 1107290592 Equivalent to 65 /8s, 255 /16s and 233 /24s ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106 (pre-ERX allocations) 2138-2584, 2615-2772, 2823-2829, 2880-3153 3354-4607, 4865-5119, 5632-6655, 6912-7466 7723-8191, 10240-12287, 13312-15359, 16384-17407 18432-20479, 21504-23551, 25600-26591, 26624-27647, 29696-30719, 31744-33791 35840-36863, 39936-40959, 46080-47103 53248-55295, 62464-63487, 64198-64296, 393216-397212 ARIN Address Blocks 3/8, 4/8, 6/8, 7/8, 8/8, 9/8, 11/8, 12/8, 13/8, 15/8, 16/8, 17/8, 18/8, 19/8, 20/8, 21/8, 22/8, 23/8, 24/8, 26/8, 28/8, 29/8, 30/8, 32/8, 33/8, 34/8, 35/8, 38/8, 40/8, 44/8, 45/8, 47/8, 48/8, 50/8, 52/8, 53/8, 54/8, 55/8, 56/8, 57/8, 63/8, 64/8, 65/8, 66/8, 67/8, 68/8, 69/8, 70/8, 71/8, 72/8, 73/8, 74/8, 75/8, 76/8, 96/8, 97/8, 98/8, 99/8, 100/8, 104/8, 107/8, 108/8, 128/8, 129/8, 130/8, 131/8, 132/8, 134/8, 135/8, 136/8, 137/8, 138/8, 139/8, 140/8, 142/8, 143/8, 144/8, 146/8, 147/8, 148/8, 149/8, 152/8, 155/8, 156/8, 157/8, 158/8, 159/8, 160/8, 161/8, 162/8, 164/8, 165/8, 166/8, 167/8, 168/8, 169/8, 170/8, 172/8, 173/8, 174/8, 184/8, 192/8, 198/8, 199/8, 204/8, 205/8, 206/8, 207/8, 208/8, 209/8, 214/8, 215/8, 216/8, RIPE Region Analysis Summary ---------------------------- Prefixes being announced by RIPE Region ASes: 146172 Total RIPE prefixes after maximum aggregation: 71917 RIPE Deaggregation factor: 2.03 Prefixes being announced from the RIPE address blocks: 156575 Unique aggregates announced from the RIPE address blocks: 96689 RIPE Region origin ASes present in the Internet Routing Table: 18139 RIPE Prefixes per ASN: 8.63 RIPE Region origin ASes announcing only one prefix: 7833 RIPE Region transit ASes present in the Internet Routing Table: 3037 Average RIPE Region AS path length visible: 4.3 Max RIPE Region AS path length visible: 27 Number of RIPE region 32-bit ASNs visible in the Routing Table: 5054 Number of RIPE addresses announced to Internet: 708935680 Equivalent to 42 /8s, 65 /16s and 128 /24s RIPE AS Blocks 1877-1901, 2043, 2047, 2107-2136, 2585-2614 (pre-ERX allocations) 2773-2822, 2830-2879, 3154-3353, 5377-5631 6656-6911, 8192-9215, 12288-13311, 15360-16383 20480-21503, 24576-25599, 28672-29695 30720-31743, 33792-35839, 38912-39935 40960-45055, 47104-52223, 56320-58367 59392-61439, 61952-62463, 64396-64495 196608-207259 RIPE Address Blocks 2/8, 5/8, 25/8, 31/8, 37/8, 46/8, 51/8, 62/8, 77/8, 78/8, 79/8, 80/8, 81/8, 82/8, 83/8, 84/8, 85/8, 86/8, 87/8, 88/8, 89/8, 90/8, 91/8, 92/8, 93/8, 94/8, 95/8, 109/8, 141/8, 145/8, 151/8, 176/8, 178/8, 185/8, 188/8, 193/8, 194/8, 195/8, 212/8, 213/8, 217/8, LACNIC Region Analysis Summary ------------------------------ Prefixes being announced by LACNIC Region ASes: 62128 Total LACNIC prefixes after maximum aggregation: 12548 LACNIC Deaggregation factor: 4.95 Prefixes being announced from the LACNIC address blocks: 78069 Unique aggregates announced from the LACNIC address blocks: 37582 LACNIC Region origin ASes present in the Internet Routing Table: 2486 LACNIC Prefixes per ASN: 31.40 LACNIC Region origin ASes announcing only one prefix: 550 LACNIC Region transit ASes present in the Internet Routing Table: 581 Average LACNIC Region AS path length visible: 4.8 Max LACNIC Region AS path length visible: 22 Number of LACNIC region 32-bit ASNs visible in the Routing Table: 2825 Number of LACNIC addresses announced to Internet: 170784832 Equivalent to 10 /8s, 45 /16s and 248 /24s LACNIC AS Blocks 26592-26623, 27648-28671, 52224-53247, 61440-61951, 64099-64197, 262144-266652 + ERX transfers LACNIC Address Blocks 177/8, 179/8, 181/8, 186/8, 187/8, 189/8, 190/8, 191/8, 200/8, 201/8, AfriNIC Region Analysis Summary ------------------------------- Prefixes being announced by AfriNIC Region ASes: 14759 Total AfriNIC prefixes after maximum aggregation: 3304 AfriNIC Deaggregation factor: 4.47 Prefixes being announced from the AfriNIC address blocks: 16836 Unique aggregates announced from the AfriNIC address blocks: 6370 AfriNIC Region origin ASes present in the Internet Routing Table: 736 AfriNIC Prefixes per ASN: 22.88 AfriNIC Region origin ASes announcing only one prefix: 174 AfriNIC Region transit ASes present in the Internet Routing Table: 179 Average AfriNIC Region AS path length visible: 4.7 Max AfriNIC Region AS path length visible: 18 Number of AfriNIC region 32-bit ASNs visible in the Routing Table: 264 Number of AfriNIC addresses announced to Internet: 82320128 Equivalent to 4 /8s, 232 /16s and 27 /24s AfriNIC AS Blocks 36864-37887, 327680-328703 & ERX transfers AfriNIC Address Blocks 41/8, 102/8, 105/8, 154/8, 196/8, 197/8, APNIC Region per AS prefix count summary ---------------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5549 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3540 382 249 TPG-INTERNET-AP TPG Telecom Limited, AU 4766 3002 11138 984 KIXS-AS-KR Korea Telecom, KR 17974 2947 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 9829 2696 1497 532 BSNL-NIB National Internet Backbone, IN 9808 2185 8781 42 CMNET-GD Guangdong Mobile Communication 4755 2055 429 218 TATACOMM-AS TATA Communications formerl 4808 1769 2289 522 CHINA169-BJ China Unicom Beijing Provin 45899 1588 1235 89 VNPT-AS-VN VNPT Corp, VN 24560 1549 516 220 AIRTELBROADBAND-AS-AP Bharti Airtel Ltd Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-APNIC ARIN Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 22773 3527 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 6389 2218 3671 41 BELLSOUTH-NET-BLK - BellSouth.net Inc., 18566 2196 405 110 MEGAPATH5-US - MegaPath Corporation, US 20115 1939 1982 405 CHARTER-NET-HKY-NC - Charter Communicat 30036 1760 340 272 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 209 1721 5071 657 CENTURYLINK-US-LEGACY-QWEST - Qwest Com 6983 1679 848 227 ITCDELTA - Earthlink, Inc., US 701 1618 10686 678 UUNET - MCI Communications Services, In 16509 1418 2537 470 AMAZON-02 - Amazon.com, Inc., US 7018 1368 20092 1008 ATT-INTERNET4 - AT&T Services, Inc., US Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-ARIN RIPE Region per AS prefix count summary --------------------------------------- ASN No of nets /20 equiv MaxAgg Description 39891 3329 169 15 ALJAWWALSTC-AS , SA 20940 2828 1073 2011 AKAMAI-ASN1 , US 34984 1972 326 356 TELLCOM-AS , TR 12479 1351 1018 46 UNI2-AS , ES 13188 1237 99 57 BANKINFORM-AS , UA 8551 1206 377 46 BEZEQ-INTERNATIONAL-AS Bezeqint Interne 6849 1144 355 21 UKRTELNET , UA 8402 1093 544 15 CORBINA-AS Russia, RU 12389 927 1113 401 ROSTELECOM-AS , RU 6830 892 2755 467 LGI-UPC formerly known as UPC Broadband Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-RIPE LACNIC Region per AS prefix count summary ----------------------------------------- ASN No of nets /20 equiv MaxAgg Description 10620 3494 541 177 Telmex Colombia S.A., CO 8151 2283 3397 555 Uninet S.A. de C.V., MX 7303 1546 957 245 Telecom Argentina S.A., AR 6503 1406 437 54 Axtel, S.A.B. de C.V., MX 11830 1324 367 57 Instituto Costarricense de Electricidad 6147 1203 377 27 Telefonica del Peru S.A.A., PE 28573 999 2213 196 CLARO S.A., BR 7738 994 1882 40 Telemar Norte Leste S.A., BR 3816 972 470 202 COLOMBIA TELECOMUNICACIONES S.A. ESP, C 11172 906 125 76 Alestra, S. de R.L. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-LACNIC AfriNIC Region per AS prefix count summary ------------------------------------------ ASN No of nets /20 equiv MaxAgg Description 24863 1184 402 50 LINKdotNET-AS, EG 36903 684 344 121 MT-MPLS, MA 37611 668 51 5 Afrihost, ZA 36992 565 1373 25 ETISALAT-MISR, EG 8452 522 1472 15 TE-AS TE-AS, EG 37492 385 252 70 ORANGE-TN, TN 24835 349 611 17 RAYA-AS, EG 29571 318 37 12 CITelecom-AS, CI 15399 298 35 7 WANANCHI-KE, KE 2018 269 327 74 TENET-1, ZA Complete listing at http://thyme.rand.apnic.net/current/data-ASnet-AFRINIC Global Per AS prefix count summary ---------------------------------- ASN No of nets /20 equiv MaxAgg Description 4538 5549 4190 74 ERX-CERNET-BKB China Education and Rese 7545 3540 382 249 TPG-INTERNET-AP TPG Telecom Limited, AU 22773 3527 2964 145 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3494 541 177 Telmex Colombia S.A., CO 39891 3329 169 15 ALJAWWALSTC-AS , SA 4766 3002 11138 984 KIXS-AS-KR Korea Telecom, KR 17974 2947 904 78 TELKOMNET-AS2-AP PT Telekomunikasi Indo 20940 2828 1073 2011 AKAMAI-ASN1 , US 9829 2696 1497 532 BSNL-NIB National Internet Backbone, IN 8151 2283 3397 555 Uninet S.A. de C.V., MX Complete listing at http://thyme.rand.apnic.net/current/data-ASnet Global Per AS Maximum Aggr summary ---------------------------------- ASN No of nets Net Savings Description 22773 3527 3382 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 10620 3494 3317 Telmex Colombia S.A., CO 39891 3329 3314 ALJAWWALSTC-AS , SA 7545 3540 3291 TPG-INTERNET-AP TPG Telecom Limited, AU 17974 2947 2869 TELKOMNET-AS2-AP PT Telekomunikasi Indo 6389 2218 2177 BELLSOUTH-NET-BLK - BellSouth.net Inc., 9829 2696 2164 BSNL-NIB National Internet Backbone, IN 9808 2185 2143 CMNET-GD Guangdong Mobile Communication 18566 2196 2086 MEGAPATH5-US - MegaPath Corporation, US 4766 3002 2018 KIXS-AS-KR Korea Telecom, KR Complete listing at http://thyme.rand.apnic.net/current/data-CIDRnet List of Unregistered Origin ASNs (Global) ----------------------------------------- Bad AS Designation Network Transit AS Description 65001 PRIVATE 5.143.176.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/20 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.192.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.200.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.208.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65001 PRIVATE 31.172.216.0/21 15468 KLGELECS-AS 38, Teatralnaya st 65000 PRIVATE 31.219.177.0/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65000 PRIVATE 31.219.177.128/25 8966 ETISALAT-AS P.O. Box 1150, Dub 65512 PRIVATE 45.252.244.0/24 45899 VNPT-AS-VN VNPT Corp, VN 65512 PRIVATE 45.252.245.0/24 45899 VNPT-AS-VN VNPT Corp, VN Complete listing at http://thyme.rand.apnic.net/current/data-badAS Advertised Unallocated Addresses -------------------------------- Network Origin AS Description 23.249.144.0/20 40430 COLO4JAX-AS - colo4jax, LLC, US 27.100.7.0/24 56096 UNKNOWN 41.73.1.0/24 37004 -Reserved AS-, ZZ 41.73.2.0/24 37004 -Reserved AS-, ZZ 41.73.3.0/24 37004 -Reserved AS-, ZZ 41.73.4.0/24 37004 -Reserved AS-, ZZ 41.73.5.0/24 37004 -Reserved AS-, ZZ 41.73.6.0/24 37004 -Reserved AS-, ZZ 41.73.7.0/24 37004 -Reserved AS-, ZZ 41.73.8.0/24 37004 -Reserved AS-, ZZ Complete listing at http://thyme.rand.apnic.net/current/data-add-IANA Number of prefixes announced per prefix length (Global) ------------------------------------------------------- /1:0 /2:0 /3:0 /4:0 /5:0 /6:0 /7:0 /8:16 /9:13 /10:36 /11:103 /12:273 /13:523 /14:1051 /15:1774 /16:13166 /17:7839 /18:13034 /19:25414 /20:38763 /21:40311 /22:68182 /23:59822 /24:340828 /25:510 /26:480 /27:339 /28:58 /29:32 /30:13 /31:1 /32:34 Advertised prefixes smaller than registry allocations ----------------------------------------------------- ASN No of nets Total ann. Description 39891 2896 3329 ALJAWWALSTC-AS , SA 22773 2748 3527 ASN-CXA-ALL-CCI-22773-RDC - Cox Communi 18566 2088 2196 MEGAPATH5-US - MegaPath Corporation, US 30036 1574 1760 MEDIACOM-ENTERPRISE-BUSINESS - Mediacom 6389 1416 2218 BELLSOUTH-NET-BLK - BellSouth.net Inc., 10620 1416 3494 Telmex Colombia S.A., CO 6983 1328 1679 ITCDELTA - Earthlink, Inc., US 34984 1257 1972 TELLCOM-AS , TR 11492 1205 1297 CABLEONE - CABLE ONE, INC., US 13188 1012 1237 BANKINFORM-AS , UA Complete listing at http://thyme.rand.apnic.net/current/data-sXXas-nos Number of /24s announced per /8 block (Global) ---------------------------------------------- 1:1641 2:797 4:23 5:2179 6:31 8:987 12:1796 13:44 14:1742 15:46 16:2 17:98 18:122 20:50 22:1 23:1860 24:1803 27:2337 31:1789 32:83 33:4 35:5 36:329 37:2322 38:1258 39:36 40:103 41:2945 42:456 43:1854 44:46 45:2144 46:2554 47:99 49:1210 50:909 51:13 52:553 53:2 54:348 55:7 56:7 57:41 58:1571 59:1014 60:376 61:1840 62:1517 63:1920 64:4565 65:2174 66:4271 67:2207 68:1123 69:3257 70:1288 71:558 72:2071 74:2582 75:401 76:414 77:1386 78:1221 79:920 80:1301 81:1404 82:988 83:705 84:857 85:1567 86:479 87:1134 88:546 89:2081 90:225 91:6093 92:950 93:2361 94:2436 95:2503 96:577 97:344 98:958 99:89 100:148 101:1153 103:12422 104:2532 105:108 106:437 107:1437 108:779 109:2254 110:1280 111:1635 112:1069 113:1143 114:1081 115:1686 116:1675 117:1567 118:2043 119:1582 120:938 121:1118 122:2264 123:2034 124:1575 125:1822 128:685 129:439 130:418 131:1374 132:622 133:175 134:489 135:197 136:387 137:399 138:1888 139:434 140:600 141:452 142:663 143:962 144:732 145:164 146:956 147:649 148:1393 149:537 150:659 151:897 152:670 153:300 154:696 155:956 156:530 157:525 158:390 159:1241 160:505 161:735 162:2408 163:568 164:774 165:1115 166:336 167:1195 168:2224 169:673 170:1969 171:252 172:719 173:1743 174:768 175:673 176:1726 177:3994 178:2249 179:1172 180:2161 181:1888 182:2078 183:1004 184:830 185:7548 186:3161 187:2124 188:2207 189:1758 190:7771 191:1298 192:9030 193:5736 194:4449 195:3839 196:1778 197:1161 198:5590 199:5749 200:7157 201:3684 202:10122 203:9840 204:4527 205:2737 206:3000 207:3078 208:4059 209:3862 210:3881 211:2044 212:2719 213:2347 214:870 215:68 216:5743 217:1991 218:802 219:611 220:1635 221:880 222:698 223:1304 End of report From bill at herrin.us Fri Sep 30 18:57:33 2016 From: bill at herrin.us (William Herrin) Date: Fri, 30 Sep 2016 14:57:33 -0400 Subject: ARIN legacy block transfer process In-Reply-To: <20160930174745.GF12280@ernw.de> References: <20c65b90-59d6-6e48-caae-8aa58985f0e7@rollernet.us> <20160930174745.GF12280@ernw.de> Message-ID: On Fri, Sep 30, 2016 at 1:47 PM, Enno Rey wrote: > Note also there's voices recommending not to sign an RSA for legacy space (in certain situations, at least), see http://ipv4marketgroup.com/dont-sign-an-rsa-during-your-82-ipv4-transfer/. Hi Enno, The article says: "Never sign an RSA as part of bringing your registry entry up to date, unless you are in the process of a transfer for an IPv4 sale" I agree with that statement. But, the situation here *IS* a transfer for an IPv4 sale. If you want to do the transfer, the originating registration has to be under a registration services agreement (RSA). As a legacy registrant, the LRSA is slightly more advantageous than the regular RSA. If the legacy registrant is actually in Europe, you might get away with bypassing ARIN. I wouldn't try it if they aren't. Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From bill at herrin.us Fri Sep 30 19:05:11 2016 From: bill at herrin.us (William Herrin) Date: Fri, 30 Sep 2016 15:05:11 -0400 Subject: ARIN legacy block transfer process In-Reply-To: <8c41e580-9d97-c919-41bb-c88ff2684a84@bryanfields.net> References: <8c41e580-9d97-c919-41bb-c88ff2684a84@bryanfields.net> Message-ID: On Fri, Sep 30, 2016 at 1:34 PM, Bryan Fields wrote: > On 9/30/16 1:22 PM, William Herrin wrote: >> Note that you can't sell the block as an "owned asset" and have ARIN >> recognize the change. ARIN does not recognize ownership of IP address >> blocks, they only recognize registration and authorized agents. > > This would seem to be in violation of what the NSF has said about this space. > I thought ARIN was slapped hard once before about this very thing? To the best of my knowledge, that's not the case. Every relevant court case has ended one of two ways: 1. The addresses were revoked after the POC was (correctly) determined not currently represent the (defunct) registrant. 2. The registrant consented to place the addresses under an ARIN RSA without a judicial ruling. (e.g. Microsoft/Nortel) Regards, Bill Herrin -- William Herrin ................ herrin at dirtside.com bill at herrin.us Owner, Dirtside Systems ......... Web: From matthew at matthew.at Fri Sep 30 19:15:29 2016 From: matthew at matthew.at (Matthew Kaufman) Date: Fri, 30 Sep 2016 19:15:29 +0000 Subject: ARIN legacy block transfer process In-Reply-To: References: <8c41e580-9d97-c919-41bb-c88ff2684a84@bryanfields.net> Message-ID: But only the recipient must put them under an RSA in order to have them registered. The source need not have an RSA or LRSA for their legacy blocks, at least as I understand it. I'd also suggest that having a broker is useful, because the few well-known ones that exist are well-versed in the process by now, for all types of sources and destinations. Matthew Kaufman On Fri, Sep 30, 2016 at 12:08 PM William Herrin wrote: > On Fri, Sep 30, 2016 at 1:34 PM, Bryan Fields > wrote: > > On 9/30/16 1:22 PM, William Herrin wrote: > >> Note that you can't sell the block as an "owned asset" and have ARIN > >> recognize the change. ARIN does not recognize ownership of IP address > >> blocks, they only recognize registration and authorized agents. > > > > This would seem to be in violation of what the NSF has said about this > space. > > I thought ARIN was slapped hard once before about this very thing? > > To the best of my knowledge, that's not the case. Every relevant court > case has ended one of two ways: > > 1. The addresses were revoked after the POC was (correctly) determined > not currently represent the (defunct) registrant. > 2. The registrant consented to place the addresses under an ARIN RSA > without a judicial ruling. (e.g. Microsoft/Nortel) > > Regards, > Bill Herrin > > > -- > William Herrin ................ herrin at dirtside.com bill at herrin.us > Owner, Dirtside Systems ......... Web: > From piotr.1234 at interia.pl Fri Sep 30 19:42:37 2016 From: piotr.1234 at interia.pl (Pedro) Date: Fri, 30 Sep 2016 21:42:37 +0200 Subject: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos Message-ID: Hello, I have some idea to put switch before bgp router in order to terminate isp 10G uplinks on switch, not router. Main reason is that could be some kind of 1st level of defence against ddos, second reason, less important, save cost of router ports, do many port mirrors. I think about N3K-C3064PQ or Juniper ex4500 because there are quite cheap and a lot of on Ebay. I would like on nexus or juniper try use some feature: - limit udp, icmp, bum packets (bandwith,pps) at ingress tagged port or vlan - create counters: passed and dropped packets, best way to get this counters via snmp oid, sent snmp traps, syslog etc in order to monitor or even as a action shut down port - port mirror from many ports/vlans to multiple port (other anty ddos solutions) - limited bgp but with flowspec to comunicate with another anty ddos devices I'm also wondering how this feature above impact on cpu/whole switch. It can be some performance degradation ot all of this feature are done in hardware, with wirespeeed ? Which model will better to do this ? Thanks for any advice, Pedro --- Ta wiadomo?? zosta?a sprawdzona na obecno?? wirus?w przez oprogramowanie antywirusowe Avast. https://www.avast.com/antivirus From saku at ytti.fi Fri Sep 30 20:06:13 2016 From: saku at ytti.fi (Saku Ytti) Date: Fri, 30 Sep 2016 23:06:13 +0300 Subject: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos In-Reply-To: References: Message-ID: On 30 September 2016 at 22:42, Pedro wrote: Hey Pedro, > I have some idea to put switch before bgp router in order to terminate isp > 10G uplinks on switch, not router. Main reason is that could be some kind of > 1st level of defence against ddos, second reason, less important, save cost > of router ports, do many port mirrors. I don't understand your rationale, unless your router is software box, but as it has 10G interface, probably not. Your router should be able to limit packets in HW, likely with better counter and filtering options than cheap switch. -- ++ytti From mlfreita at mtu.edu Fri Sep 30 20:50:25 2016 From: mlfreita at mtu.edu (Matt Freitag) Date: Fri, 30 Sep 2016 16:50:25 -0400 Subject: nexus N3K-C3064PQ vs juniper ex4500 in order to protect against ddos In-Reply-To: References: Message-ID: Pedro, Please also keep in mind that the Juniper EX4500 is an end of life product. Soon you won't be able to get Juniper to support you. That's why there are so many for so cheap on eBay. Matt Freitag Network Engineer I Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.it.mtu.edu/ On Fri, Sep 30, 2016 at 4:06 PM, Saku Ytti wrote: > On 30 September 2016 at 22:42, Pedro wrote: > > Hey Pedro, > > > I have some idea to put switch before bgp router in order to terminate > isp > > 10G uplinks on switch, not router. Main reason is that could be some > kind of > > 1st level of defence against ddos, second reason, less important, save > cost > > of router ports, do many port mirrors. > > I don't understand your rationale, unless your router is software box, > but as it has 10G interface, probably not. > Your router should be able to limit packets in HW, likely with better > counter and filtering options than cheap switch. > > -- > ++ytti >